2026-03-31T20:58:15.720 INFO:root:teuthology version: 1.2.4.dev37+ga59626679 2026-03-31T20:58:15.725 DEBUG:teuthology.report:Pushing job info to http://localhost:8080 2026-03-31T20:58:15.745 INFO:teuthology.run:Config: archive_path: /archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348 branch: tentacle description: rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{avl} base mem$/{normal-2} onode-segment$/{256K} write$/{random/{compr$/{no$/{no}} random}}}} rados supported-random-distro$/{centos_latest}} email: null first_in_suite: false flavor: default job_id: '4348' last_in_suite: false machine_type: vps name: kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: tentacle ansible.cephlab: branch: main repo: https://github.com/kshtsk/ceph-cm-ansible.git skip_tags: nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs vars: logical_volumes: lv_1: scratch_dev: true size: 25%VG vg: vg_nvme lv_2: scratch_dev: true size: 25%VG vg: vg_nvme lv_3: scratch_dev: true size: 25%VG vg: vg_nvme lv_4: scratch_dev: true size: 25%VG vg: vg_nvme timezone: UTC volume_groups: vg_nvme: pvs: /dev/vdb,/dev/vdc,/dev/vdd,/dev/vde ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 3 mon mgr beacon grace: 90 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: avl bluestore block size: 96636764160 bluestore fsck on mount: true bluestore onode segment size: 256K bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock iops capacity threshold hdd: 49000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd mclock skip benchmark: true osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 5bb3278730741031382ca9c3dc9d221a942e06a2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_binary_url: https://download.ceph.com/rpm-20.2.0/el9/noarch/cephadm install: ceph: flavor: default sha1: 5bb3278730741031382ca9c3dc9d221a942e06a2 extra_system_packages: deb: - python3-jmespath - python3-xmltodict - s3cmd rpm: - bzip2 - perl-Test-Harness - python3-jmespath - python3-xmltodict - s3cmd thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: tt-tentacle sha1: 0392f78529848ec72469e8e431875cb98d3a5fb4 owner: kyr priority: 1000 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 6407 sha1: 5bb3278730741031382ca9c3dc9d221a942e06a2 sleep_before_teardown: 0 subset: 1/100000 suite: rados suite_branch: tt-tentacle suite_path: /home/teuthos/src/github.com_kshtsk_ceph_0392f78529848ec72469e8e431875cb98d3a5fb4/qa suite_relpath: qa suite_repo: https://github.com/kshtsk/ceph.git suite_sha1: 0392f78529848ec72469e8e431875cb98d3a5fb4 targets: vm00.local: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnNB2WMelB5Hnf3uHsc711hbb+KoMw3iMoXVqaJL51aDJu0TdvEFEuQE8CKQEmUxO8zosMwhwGoucJq8WTttG8= tasks: - install: null - ceph: conf: osd: osd max pg log entries: 5 osd min pg log entries: 5 create_rbd_pool: false log-ignorelist: - \(OBJECT_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SLOW_OPS\) - deep-scrub - missing - overall HEALTH_ - repair - slow request - unfound - \(POOL_APP_NOT_ENABLED\) - enough copies available pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - ec_inconsistent_hinfo: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: uv2 teuthology_repo: https://github.com/kshtsk/teuthology teuthology_sha1: a59626679648f962bca99d20d35578f2998c8f37 timestamp: 2026-03-31_11:18:10 tube: vps user: kyr verbose: false worker_log: /home/teuthos/.teuthology/dispatcher/dispatcher.vps.282426 2026-03-31T20:58:15.745 INFO:teuthology.run:suite_path is set to /home/teuthos/src/github.com_kshtsk_ceph_0392f78529848ec72469e8e431875cb98d3a5fb4/qa; will attempt to use it 2026-03-31T20:58:15.745 INFO:teuthology.run:Found tasks at /home/teuthos/src/github.com_kshtsk_ceph_0392f78529848ec72469e8e431875cb98d3a5fb4/qa/tasks 2026-03-31T20:58:15.745 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-03-31T20:58:15.746 INFO:teuthology.task.internal:Checking packages... 2026-03-31T20:58:15.746 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5bb3278730741031382ca9c3dc9d221a942e06a2' 2026-03-31T20:58:15.746 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-03-31T20:58:15.746 INFO:teuthology.packaging:ref: None 2026-03-31T20:58:15.746 INFO:teuthology.packaging:tag: None 2026-03-31T20:58:15.746 INFO:teuthology.packaging:branch: tentacle 2026-03-31T20:58:15.746 INFO:teuthology.packaging:sha1: 5bb3278730741031382ca9c3dc9d221a942e06a2 2026-03-31T20:58:15.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=tentacle 2026-03-31T20:58:16.526 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-721.g5bb32787 2026-03-31T20:58:16.526 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-03-31T20:58:16.527 INFO:teuthology.task.internal:no buildpackages task found 2026-03-31T20:58:16.527 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-03-31T20:58:16.527 INFO:teuthology.task.internal:Saving configuration 2026-03-31T20:58:16.533 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-03-31T20:58:16.534 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-03-31T20:58:16.540 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'vm00.local', 'description': '/archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348', 'up': True, 'machine_type': 'vps', 'is_vm': True, 'vm_host': {'name': 'localhost', 'description': None, 'up': True, 'machine_type': 'libvirt', 'is_vm': False, 'vm_host': None, 'os_type': None, 'os_version': None, 'arch': None, 'locked': True, 'locked_since': None, 'locked_by': None, 'mac_address': None, 'ssh_pub_key': None}, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-03-31 20:57:38.836942', 'locked_by': 'kyr', 'mac_address': '52:55:00:00:00:00', 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnNB2WMelB5Hnf3uHsc711hbb+KoMw3iMoXVqaJL51aDJu0TdvEFEuQE8CKQEmUxO8zosMwhwGoucJq8WTttG8='} 2026-03-31T20:58:16.540 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-03-31T20:58:16.540 INFO:teuthology.task.internal:roles: ubuntu@vm00.local - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2026-03-31T20:58:16.540 INFO:teuthology.run_tasks:Running task console_log... 2026-03-31T20:58:16.546 DEBUG:teuthology.task.console_log:vm00 does not support IPMI; excluding 2026-03-31T20:58:16.546 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4aed99e8c0>, signals=[15]) 2026-03-31T20:58:16.546 INFO:teuthology.run_tasks:Running task internal.connect... 2026-03-31T20:58:16.547 INFO:teuthology.task.internal:Opening connections... 2026-03-31T20:58:16.547 DEBUG:teuthology.task.internal:connecting to ubuntu@vm00.local 2026-03-31T20:58:16.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm00.local', 'username': 'ubuntu', 'timeout': 60} 2026-03-31T20:58:16.625 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-03-31T20:58:16.627 DEBUG:teuthology.orchestra.run.vm00:> uname -m 2026-03-31T20:58:16.785 INFO:teuthology.orchestra.run.vm00.stdout:x86_64 2026-03-31T20:58:16.786 DEBUG:teuthology.orchestra.run.vm00:> cat /etc/os-release 2026-03-31T20:58:16.840 INFO:teuthology.orchestra.run.vm00.stdout:NAME="CentOS Stream" 2026-03-31T20:58:16.840 INFO:teuthology.orchestra.run.vm00.stdout:VERSION="9" 2026-03-31T20:58:16.840 INFO:teuthology.orchestra.run.vm00.stdout:ID="centos" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:ID_LIKE="rhel fedora" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:VERSION_ID="9" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:PLATFORM_ID="platform:el9" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:PRETTY_NAME="CentOS Stream 9" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:ANSI_COLOR="0;31" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:LOGO="fedora-logo-icon" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:HOME_URL="https://centos.org/" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-03-31T20:58:16.841 INFO:teuthology.orchestra.run.vm00.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-03-31T20:58:16.841 INFO:teuthology.lock.ops:Updating vm00.local on lock server 2026-03-31T20:58:16.846 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-03-31T20:58:16.847 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-03-31T20:58:16.848 INFO:teuthology.task.internal:Checking for old test directory... 2026-03-31T20:58:16.848 DEBUG:teuthology.orchestra.run.vm00:> test '!' -e /home/ubuntu/cephtest 2026-03-31T20:58:16.894 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-03-31T20:58:16.895 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-03-31T20:58:16.895 DEBUG:teuthology.orchestra.run.vm00:> test -z $(ls -A /var/lib/ceph) 2026-03-31T20:58:16.949 INFO:teuthology.orchestra.run.vm00.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-03-31T20:58:16.950 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-03-31T20:58:16.958 DEBUG:teuthology.orchestra.run.vm00:> test -e /ceph-qa-ready 2026-03-31T20:58:17.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-03-31T20:58:17.225 INFO:teuthology.run_tasks:Running task internal.base... 2026-03-31T20:58:17.226 INFO:teuthology.task.internal:Creating test directory... 2026-03-31T20:58:17.226 DEBUG:teuthology.orchestra.run.vm00:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-03-31T20:58:17.240 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-03-31T20:58:17.241 INFO:teuthology.run_tasks:Running task internal.archive... 2026-03-31T20:58:17.242 INFO:teuthology.task.internal:Creating archive directory... 2026-03-31T20:58:17.242 DEBUG:teuthology.orchestra.run.vm00:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-03-31T20:58:17.297 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-03-31T20:58:17.299 INFO:teuthology.task.internal:Enabling coredump saving... 2026-03-31T20:58:17.299 DEBUG:teuthology.orchestra.run.vm00:> test -f /run/.containerenv -o -f /.dockerenv 2026-03-31T20:58:17.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-03-31T20:58:17.351 DEBUG:teuthology.orchestra.run.vm00:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-03-31T20:58:17.414 INFO:teuthology.orchestra.run.vm00.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-03-31T20:58:17.423 INFO:teuthology.orchestra.run.vm00.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-03-31T20:58:17.424 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-03-31T20:58:17.425 INFO:teuthology.task.internal:Configuring sudo... 2026-03-31T20:58:17.425 DEBUG:teuthology.orchestra.run.vm00:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-03-31T20:58:17.487 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-03-31T20:58:17.489 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-03-31T20:58:17.489 DEBUG:teuthology.orchestra.run.vm00:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-03-31T20:58:17.541 DEBUG:teuthology.orchestra.run.vm00:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-03-31T20:58:17.603 DEBUG:teuthology.orchestra.run.vm00:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-03-31T20:58:17.657 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T20:58:17.658 DEBUG:teuthology.orchestra.run.vm00:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-03-31T20:58:17.717 DEBUG:teuthology.orchestra.run.vm00:> sudo service rsyslog restart 2026-03-31T20:58:17.783 INFO:teuthology.orchestra.run.vm00.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-03-31T20:58:18.145 INFO:teuthology.run_tasks:Running task internal.timer... 2026-03-31T20:58:18.147 INFO:teuthology.task.internal:Starting timer... 2026-03-31T20:58:18.147 INFO:teuthology.run_tasks:Running task pcp... 2026-03-31T20:58:18.149 INFO:teuthology.run_tasks:Running task selinux... 2026-03-31T20:58:18.151 INFO:teuthology.task.selinux:Excluding vm00: VMs are not yet supported 2026-03-31T20:58:18.151 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-03-31T20:58:18.151 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-03-31T20:58:18.152 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-03-31T20:58:18.152 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-03-31T20:58:18.153 DEBUG:teuthology.task:Applying overrides for task ansible.cephlab: {'branch': 'main', 'repo': 'https://github.com/kshtsk/ceph-cm-ansible.git', 'skip_tags': 'nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs', 'vars': {'logical_volumes': {'lv_1': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_2': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_3': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_4': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}}, 'timezone': 'UTC', 'volume_groups': {'vg_nvme': {'pvs': '/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde'}}}} 2026-03-31T20:58:18.153 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthos/src/github.com_kshtsk_ceph-cm-ansible_main to origin/main 2026-03-31T20:58:18.159 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-03-31T20:58:18.159 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu", "logical_volumes": {"lv_1": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_2": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_3": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}, "lv_4": {"scratch_dev": true, "size": "25%VG", "vg": "vg_nvme"}}, "timezone": "UTC", "volume_groups": {"vg_nvme": {"pvs": "/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde"}}}' -i /tmp/teuth_ansible_inventoryolmpmc0a --limit vm00.local /home/teuthos/src/github.com_kshtsk_ceph-cm-ansible_main/cephlab.yml --skip-tags nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs 2026-03-31T20:59:46.166 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@vm00.local')] 2026-03-31T20:59:46.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@vm00.local' 2026-03-31T20:59:46.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'vm00.local', 'username': 'ubuntu', 'timeout': 60} 2026-03-31T20:59:46.233 DEBUG:teuthology.orchestra.run.vm00:> true 2026-03-31T20:59:46.310 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@vm00.local' 2026-03-31T20:59:46.310 INFO:teuthology.run_tasks:Running task clock... 2026-03-31T20:59:46.312 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-03-31T20:59:46.312 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-03-31T20:59:46.313 DEBUG:teuthology.orchestra.run.vm00:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-03-31T20:59:46.384 INFO:teuthology.orchestra.run.vm00.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-03-31T20:59:46.399 INFO:teuthology.orchestra.run.vm00.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-03-31T20:59:46.425 INFO:teuthology.orchestra.run.vm00.stderr:sudo: ntpd: command not found 2026-03-31T20:59:46.438 INFO:teuthology.orchestra.run.vm00.stdout:506 Cannot talk to daemon 2026-03-31T20:59:46.453 INFO:teuthology.orchestra.run.vm00.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-03-31T20:59:46.471 INFO:teuthology.orchestra.run.vm00.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-03-31T20:59:46.521 INFO:teuthology.orchestra.run.vm00.stderr:bash: line 1: ntpq: command not found 2026-03-31T20:59:46.524 INFO:teuthology.orchestra.run.vm00.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-03-31T20:59:46.524 INFO:teuthology.orchestra.run.vm00.stdout:=============================================================================== 2026-03-31T20:59:46.524 INFO:teuthology.run_tasks:Running task install... 2026-03-31T20:59:46.526 DEBUG:teuthology.task.install:project ceph 2026-03-31T20:59:46.526 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5bb3278730741031382ca9c3dc9d221a942e06a2'}, 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}} 2026-03-31T20:59:46.526 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5bb3278730741031382ca9c3dc9d221a942e06a2', 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}} 2026-03-31T20:59:46.526 INFO:teuthology.task.install:Using flavor: default 2026-03-31T20:59:46.528 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-03-31T20:59:46.528 INFO:teuthology.task.install:extra packages: [] 2026-03-31T20:59:46.529 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5bb3278730741031382ca9c3dc9d221a942e06a2', 'tag': None, 'wait_for_package': False} 2026-03-31T20:59:46.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5bb3278730741031382ca9c3dc9d221a942e06a2 2026-03-31T20:59:47.150 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/tentacle-release/5bb3278730741031382ca9c3dc9d221a942e06a2/centos/9/flavors/default/ 2026-03-31T20:59:47.151 INFO:teuthology.task.install.rpm:Package version is 20.2.0-721.g5bb32787 2026-03-31T20:59:47.686 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/tentacle-release/5bb3278730741031382ca9c3dc9d221a942e06a2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/tentacle-release/5bb3278730741031382ca9c3dc9d221a942e06a2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/tentacle-release/5bb3278730741031382ca9c3dc9d221a942e06a2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-03-31T20:59:47.687 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T20:59:47.687 DEBUG:teuthology.orchestra.run.vm00:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-03-31T20:59:47.717 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, bzip2, perl-Test-Harness, python3-jmespath, python3-xmltodict, s3cmd on remote rpm x86_64 2026-03-31T20:59:47.718 DEBUG:teuthology.orchestra.run.vm00:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' /etc/yum.repos.d/ceph.repo ; fi 2026-03-31T20:59:47.784 DEBUG:teuthology.orchestra.run.vm00:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-03-31T20:59:47.857 DEBUG:teuthology.orchestra.run.vm00:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-03-31T20:59:47.926 INFO:teuthology.orchestra.run.vm00.stdout:check_obsoletes = 1 2026-03-31T20:59:47.928 DEBUG:teuthology.orchestra.run.vm00:> sudo yum clean all 2026-03-31T20:59:48.112 INFO:teuthology.orchestra.run.vm00.stdout:41 files removed 2026-03-31T20:59:48.140 DEBUG:teuthology.orchestra.run.vm00:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd bzip2 perl-Test-Harness python3-jmespath python3-xmltodict s3cmd 2026-03-31T20:59:49.574 INFO:teuthology.orchestra.run.vm00.stdout:ceph packages for x86_64 70 kB/s | 89 kB 00:01 2026-03-31T20:59:50.634 INFO:teuthology.orchestra.run.vm00.stdout:ceph noarch packages 18 kB/s | 19 kB 00:01 2026-03-31T20:59:51.608 INFO:teuthology.orchestra.run.vm00.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2026-03-31T20:59:52.853 INFO:teuthology.orchestra.run.vm00.stdout:CentOS Stream 9 - BaseOS 7.3 MB/s | 8.9 MB 00:01 2026-03-31T20:59:54.425 INFO:teuthology.orchestra.run.vm00.stdout:CentOS Stream 9 - AppStream 28 MB/s | 27 MB 00:00 2026-03-31T20:59:58.568 INFO:teuthology.orchestra.run.vm00.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 8.0 MB 00:01 2026-03-31T20:59:59.749 INFO:teuthology.orchestra.run.vm00.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 21 kB 00:00 2026-03-31T21:00:00.642 INFO:teuthology.orchestra.run.vm00.stdout:Extra Packages for Enterprise Linux 25 MB/s | 20 MB 00:00 2026-03-31T21:00:05.108 INFO:teuthology.orchestra.run.vm00.stdout:lab-extras 65 kB/s | 50 kB 00:00 2026-03-31T21:00:06.455 INFO:teuthology.orchestra.run.vm00.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2026-03-31T21:00:06.455 INFO:teuthology.orchestra.run.vm00.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2026-03-31T21:00:06.487 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:00:06.491 INFO:teuthology.orchestra.run.vm00.stdout:====================================================================================== 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout:====================================================================================== 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout:Installing: 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: bzip2 x86_64 1.0.8-11.el9 baseos 55 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph x86_64 2:20.2.0-721.g5bb32787.el9 ceph 6.5 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-base x86_64 2:20.2.0-721.g5bb32787.el9 ceph 5.9 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-fuse x86_64 2:20.2.0-721.g5bb32787.el9 ceph 940 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-721.g5bb32787.el9 ceph 154 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr x86_64 2:20.2.0-721.g5bb32787.el9 ceph 961 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-cephadm noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 173 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-dashboard noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 11 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 7.4 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-rook noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 50 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-radosgw x86_64 2:20.2.0-721.g5bb32787.el9 ceph 24 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-test x86_64 2:20.2.0-721.g5bb32787.el9 ceph 84 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-volume noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 298 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: cephadm noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 1.0 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-devel x86_64 2:20.2.0-721.g5bb32787.el9 ceph 34 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs2 x86_64 2:20.2.0-721.g5bb32787.el9 ceph 867 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: librados-devel x86_64 2:20.2.0-721.g5bb32787.el9 ceph 126 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: perl-Test-Harness noarch 1:3.42-461.el9 appstream 295 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: python3-cephfs x86_64 2:20.2.0-721.g5bb32787.el9 ceph 163 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: python3-rados x86_64 2:20.2.0-721.g5bb32787.el9 ceph 323 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: python3-rbd x86_64 2:20.2.0-721.g5bb32787.el9 ceph 304 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: python3-rgw x86_64 2:20.2.0-721.g5bb32787.el9 ceph 99 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: rbd-fuse x86_64 2:20.2.0-721.g5bb32787.el9 ceph 91 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: rbd-mirror x86_64 2:20.2.0-721.g5bb32787.el9 ceph 2.9 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: rbd-nbd x86_64 2:20.2.0-721.g5bb32787.el9 ceph 179 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: s3cmd noarch 2.4.0-1.el9 epel 206 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout:Upgrading: 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: librados2 x86_64 2:20.2.0-721.g5bb32787.el9 ceph 3.5 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: librbd1 x86_64 2:20.2.0-721.g5bb32787.el9 ceph 2.8 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout:Installing dependencies: 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-common x86_64 2:20.2.0-721.g5bb32787.el9 ceph 24 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-grafana-dashboards noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 43 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mds x86_64 2:20.2.0-721.g5bb32787.el9 ceph 2.3 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-modules-core noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 290 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mon x86_64 2:20.2.0-721.g5bb32787.el9 ceph 5.0 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-osd x86_64 2:20.2.0-721.g5bb32787.el9 ceph 17 M 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-prometheus-alerts noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 17 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: ceph-selinux x86_64 2:20.2.0-721.g5bb32787.el9 ceph 25 k 2026-03-31T21:00:06.492 INFO:teuthology.orchestra.run.vm00.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: fuse x86_64 2.9.9-17.el9 baseos 80 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-proxy2 x86_64 2:20.2.0-721.g5bb32787.el9 ceph 24 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libcephsqlite x86_64 2:20.2.0-721.g5bb32787.el9 ceph 164 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libradosstriper1 x86_64 2:20.2.0-721.g5bb32787.el9 ceph 250 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: librgw2 x86_64 2:20.2.0-721.g5bb32787.el9 ceph 6.4 M 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: perl-Benchmark noarch 1.23-483.el9 appstream 26 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-argparse x86_64 2:20.2.0-721.g5bb32787.el9 ceph 45 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-common x86_64 2:20.2.0-721.g5bb32787.el9 ceph 175 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-03-31T21:00:06.493 INFO:teuthology.orchestra.run.vm00.stdout: python3-cheroot noarch 10.0.1-5.el9 epel 173 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-cherrypy noarch 18.10.0-5.el9 epel 290 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-03-31T21:00:06.494 INFO:teuthology.orchestra.run.vm00.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout:Installing weak dependencies: 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-k8sevents noarch 2:20.2.0-721.g5bb32787.el9 ceph-noarch 22 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-daemon x86_64 2:20.2.0-721.g5bb32787.el9 ceph 35 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: smartmontools x86_64 1:7.2-10.el9 baseos 556 k 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout:====================================================================================== 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout:Install 148 Packages 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout:Upgrade 2 Packages 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout:Total download size: 270 M 2026-03-31T21:00:06.495 INFO:teuthology.orchestra.run.vm00.stdout:Downloading Packages: 2026-03-31T21:00:08.434 INFO:teuthology.orchestra.run.vm00.stdout:(1/150): ceph-20.2.0-721.g5bb32787.el9.x86_64.r 13 kB/s | 6.5 kB 00:00 2026-03-31T21:00:09.274 INFO:teuthology.orchestra.run.vm00.stdout:(2/150): ceph-fuse-20.2.0-721.g5bb32787.el9.x86 1.1 MB/s | 940 kB 00:00 2026-03-31T21:00:09.397 INFO:teuthology.orchestra.run.vm00.stdout:(3/150): ceph-immutable-object-cache-20.2.0-721 1.2 MB/s | 154 kB 00:00 2026-03-31T21:00:09.478 INFO:teuthology.orchestra.run.vm00.stdout:(4/150): ceph-base-20.2.0-721.g5bb32787.el9.x86 3.9 MB/s | 5.9 MB 00:01 2026-03-31T21:00:09.802 INFO:teuthology.orchestra.run.vm00.stdout:(5/150): ceph-mds-20.2.0-721.g5bb32787.el9.x86_ 5.8 MB/s | 2.3 MB 00:00 2026-03-31T21:00:10.527 INFO:teuthology.orchestra.run.vm00.stdout:(6/150): ceph-mon-20.2.0-721.g5bb32787.el9.x86_ 7.0 MB/s | 5.0 MB 00:00 2026-03-31T21:00:10.583 INFO:teuthology.orchestra.run.vm00.stdout:(7/150): ceph-common-20.2.0-721.g5bb32787.el9.x 9.0 MB/s | 24 MB 00:02 2026-03-31T21:00:11.017 INFO:teuthology.orchestra.run.vm00.stdout:(8/150): ceph-mgr-20.2.0-721.g5bb32787.el9.x86_ 625 kB/s | 961 kB 00:01 2026-03-31T21:00:11.134 INFO:teuthology.orchestra.run.vm00.stdout:(9/150): ceph-selinux-20.2.0-721.g5bb32787.el9. 215 kB/s | 25 kB 00:00 2026-03-31T21:00:12.154 INFO:teuthology.orchestra.run.vm00.stdout:(10/150): ceph-radosgw-20.2.0-721.g5bb32787.el9 15 MB/s | 24 MB 00:01 2026-03-31T21:00:12.270 INFO:teuthology.orchestra.run.vm00.stdout:(11/150): libcephfs-daemon-20.2.0-721.g5bb32787 306 kB/s | 35 kB 00:00 2026-03-31T21:00:12.386 INFO:teuthology.orchestra.run.vm00.stdout:(12/150): libcephfs-devel-20.2.0-721.g5bb32787. 296 kB/s | 34 kB 00:00 2026-03-31T21:00:12.465 INFO:teuthology.orchestra.run.vm00.stdout:(13/150): ceph-osd-20.2.0-721.g5bb32787.el9.x86 8.8 MB/s | 17 MB 00:01 2026-03-31T21:00:12.503 INFO:teuthology.orchestra.run.vm00.stdout:(14/150): libcephfs-proxy2-20.2.0-721.g5bb32787 207 kB/s | 24 kB 00:00 2026-03-31T21:00:12.594 INFO:teuthology.orchestra.run.vm00.stdout:(15/150): libcephfs2-20.2.0-721.g5bb32787.el9.x 6.5 MB/s | 867 kB 00:00 2026-03-31T21:00:12.621 INFO:teuthology.orchestra.run.vm00.stdout:(16/150): libcephsqlite-20.2.0-721.g5bb32787.el 1.4 MB/s | 164 kB 00:00 2026-03-31T21:00:12.715 INFO:teuthology.orchestra.run.vm00.stdout:(17/150): librados-devel-20.2.0-721.g5bb32787.e 1.0 MB/s | 126 kB 00:00 2026-03-31T21:00:12.740 INFO:teuthology.orchestra.run.vm00.stdout:(18/150): libradosstriper1-20.2.0-721.g5bb32787 2.1 MB/s | 250 kB 00:00 2026-03-31T21:00:12.857 INFO:teuthology.orchestra.run.vm00.stdout:(19/150): python3-ceph-argparse-20.2.0-721.g5bb 386 kB/s | 45 kB 00:00 2026-03-31T21:00:12.976 INFO:teuthology.orchestra.run.vm00.stdout:(20/150): python3-ceph-common-20.2.0-721.g5bb32 1.4 MB/s | 175 kB 00:00 2026-03-31T21:00:13.107 INFO:teuthology.orchestra.run.vm00.stdout:(21/150): python3-cephfs-20.2.0-721.g5bb32787.e 1.2 MB/s | 163 kB 00:00 2026-03-31T21:00:13.236 INFO:teuthology.orchestra.run.vm00.stdout:(22/150): python3-rados-20.2.0-721.g5bb32787.el 2.4 MB/s | 323 kB 00:00 2026-03-31T21:00:13.356 INFO:teuthology.orchestra.run.vm00.stdout:(23/150): python3-rbd-20.2.0-721.g5bb32787.el9. 2.5 MB/s | 304 kB 00:00 2026-03-31T21:00:13.474 INFO:teuthology.orchestra.run.vm00.stdout:(24/150): python3-rgw-20.2.0-721.g5bb32787.el9. 844 kB/s | 99 kB 00:00 2026-03-31T21:00:13.592 INFO:teuthology.orchestra.run.vm00.stdout:(25/150): rbd-fuse-20.2.0-721.g5bb32787.el9.x86 774 kB/s | 91 kB 00:00 2026-03-31T21:00:13.685 INFO:teuthology.orchestra.run.vm00.stdout:(26/150): librgw2-20.2.0-721.g5bb32787.el9.x86_ 6.6 MB/s | 6.4 MB 00:00 2026-03-31T21:00:13.816 INFO:teuthology.orchestra.run.vm00.stdout:(27/150): rbd-nbd-20.2.0-721.g5bb32787.el9.x86_ 1.3 MB/s | 179 kB 00:00 2026-03-31T21:00:13.847 INFO:teuthology.orchestra.run.vm00.stdout:(28/150): rbd-mirror-20.2.0-721.g5bb32787.el9.x 11 MB/s | 2.9 MB 00:00 2026-03-31T21:00:13.936 INFO:teuthology.orchestra.run.vm00.stdout:(29/150): ceph-grafana-dashboards-20.2.0-721.g5 358 kB/s | 43 kB 00:00 2026-03-31T21:00:13.964 INFO:teuthology.orchestra.run.vm00.stdout:(30/150): ceph-mgr-cephadm-20.2.0-721.g5bb32787 1.4 MB/s | 173 kB 00:00 2026-03-31T21:00:14.456 INFO:teuthology.orchestra.run.vm00.stdout:(31/150): ceph-mgr-diskprediction-local-20.2.0- 15 MB/s | 7.4 MB 00:00 2026-03-31T21:00:14.572 INFO:teuthology.orchestra.run.vm00.stdout:(32/150): ceph-mgr-k8sevents-20.2.0-721.g5bb327 191 kB/s | 22 kB 00:00 2026-03-31T21:00:14.691 INFO:teuthology.orchestra.run.vm00.stdout:(33/150): ceph-mgr-modules-core-20.2.0-721.g5bb 2.4 MB/s | 290 kB 00:00 2026-03-31T21:00:14.807 INFO:teuthology.orchestra.run.vm00.stdout:(34/150): ceph-mgr-rook-20.2.0-721.g5bb32787.el 433 kB/s | 50 kB 00:00 2026-03-31T21:00:14.923 INFO:teuthology.orchestra.run.vm00.stdout:(35/150): ceph-prometheus-alerts-20.2.0-721.g5b 150 kB/s | 17 kB 00:00 2026-03-31T21:00:15.042 INFO:teuthology.orchestra.run.vm00.stdout:(36/150): ceph-volume-20.2.0-721.g5bb32787.el9. 2.4 MB/s | 298 kB 00:00 2026-03-31T21:00:15.168 INFO:teuthology.orchestra.run.vm00.stdout:(37/150): cephadm-20.2.0-721.g5bb32787.el9.noar 7.9 MB/s | 1.0 MB 00:00 2026-03-31T21:00:15.382 INFO:teuthology.orchestra.run.vm00.stdout:(38/150): ceph-mgr-dashboard-20.2.0-721.g5bb327 7.3 MB/s | 11 MB 00:01 2026-03-31T21:00:15.421 INFO:teuthology.orchestra.run.vm00.stdout:(39/150): bzip2-1.0.8-11.el9.x86_64.rpm 217 kB/s | 55 kB 00:00 2026-03-31T21:00:15.595 INFO:teuthology.orchestra.run.vm00.stdout:(40/150): fuse-2.9.9-17.el9.x86_64.rpm 459 kB/s | 80 kB 00:00 2026-03-31T21:00:15.722 INFO:teuthology.orchestra.run.vm00.stdout:(41/150): ledmon-libs-1.1.0-3.el9.x86_64.rpm 320 kB/s | 40 kB 00:00 2026-03-31T21:00:15.731 INFO:teuthology.orchestra.run.vm00.stdout:(42/150): cryptsetup-2.8.1-3.el9.x86_64.rpm 1.0 MB/s | 351 kB 00:00 2026-03-31T21:00:15.786 INFO:teuthology.orchestra.run.vm00.stdout:(43/150): libconfig-1.7.2-9.el9.x86_64.rpm 1.1 MB/s | 72 kB 00:00 2026-03-31T21:00:15.886 INFO:teuthology.orchestra.run.vm00.stdout:(44/150): libquadmath-11.5.0-14.el9.x86_64.rpm 1.8 MB/s | 184 kB 00:00 2026-03-31T21:00:15.920 INFO:teuthology.orchestra.run.vm00.stdout:(45/150): mailcap-2.1.49-5.el9.noarch.rpm 963 kB/s | 33 kB 00:00 2026-03-31T21:00:15.970 INFO:teuthology.orchestra.run.vm00.stdout:(46/150): libgfortran-11.5.0-14.el9.x86_64.rpm 3.2 MB/s | 794 kB 00:00 2026-03-31T21:00:16.060 INFO:teuthology.orchestra.run.vm00.stdout:(47/150): pciutils-3.7.0-7.el9.x86_64.rpm 1.0 MB/s | 93 kB 00:00 2026-03-31T21:00:16.151 INFO:teuthology.orchestra.run.vm00.stdout:(48/150): nvme-cli-2.16-1.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2026-03-31T21:00:16.172 INFO:teuthology.orchestra.run.vm00.stdout:(49/150): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 253 kB 00:00 2026-03-31T21:00:16.417 INFO:teuthology.orchestra.run.vm00.stdout:(50/150): ceph-test-20.2.0-721.g5bb32787.el9.x8 16 MB/s | 84 MB 00:05 2026-03-31T21:00:16.542 INFO:teuthology.orchestra.run.vm00.stdout:(51/150): python3-ply-3.11-14.el9.noarch.rpm 288 kB/s | 106 kB 00:00 2026-03-31T21:00:16.599 INFO:teuthology.orchestra.run.vm00.stdout:(52/150): python3-pyparsing-2.4.7-9.el9.noarch. 2.6 MB/s | 150 kB 00:00 2026-03-31T21:00:16.616 INFO:teuthology.orchestra.run.vm00.stdout:(53/150): python3-cryptography-36.0.1-5.el9.x86 2.7 MB/s | 1.2 MB 00:00 2026-03-31T21:00:16.663 INFO:teuthology.orchestra.run.vm00.stdout:(54/150): python3-requests-2.25.1-10.el9.noarch 1.9 MB/s | 126 kB 00:00 2026-03-31T21:00:16.691 INFO:teuthology.orchestra.run.vm00.stdout:(55/150): python3-pycparser-2.20-6.el9.noarch.r 495 kB/s | 135 kB 00:00 2026-03-31T21:00:16.745 INFO:teuthology.orchestra.run.vm00.stdout:(56/150): python3-urllib3-1.26.5-7.el9.noarch.r 1.7 MB/s | 218 kB 00:00 2026-03-31T21:00:16.842 INFO:teuthology.orchestra.run.vm00.stdout:(57/150): unzip-6.0-59.el9.x86_64.rpm 1.2 MB/s | 182 kB 00:00 2026-03-31T21:00:16.844 INFO:teuthology.orchestra.run.vm00.stdout:(58/150): smartmontools-7.2-10.el9.x86_64.rpm 3.0 MB/s | 556 kB 00:00 2026-03-31T21:00:16.988 INFO:teuthology.orchestra.run.vm00.stdout:(59/150): flexiblas-3.0.4-9.el9.x86_64.rpm 206 kB/s | 30 kB 00:00 2026-03-31T21:00:17.083 INFO:teuthology.orchestra.run.vm00.stdout:(60/150): boost-program-options-1.75.0-13.el9.x 433 kB/s | 104 kB 00:00 2026-03-31T21:00:17.133 INFO:teuthology.orchestra.run.vm00.stdout:(61/150): flexiblas-openblas-openmp-3.0.4-9.el9 295 kB/s | 15 kB 00:00 2026-03-31T21:00:17.229 INFO:teuthology.orchestra.run.vm00.stdout:(62/150): libnbd-1.20.3-4.el9.x86_64.rpm 1.7 MB/s | 164 kB 00:00 2026-03-31T21:00:17.281 INFO:teuthology.orchestra.run.vm00.stdout:(63/150): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.0 MB/s | 160 kB 00:00 2026-03-31T21:00:17.328 INFO:teuthology.orchestra.run.vm00.stdout:(64/150): flexiblas-netlib-3.0.4-9.el9.x86_64.r 8.8 MB/s | 3.0 MB 00:00 2026-03-31T21:00:17.337 INFO:teuthology.orchestra.run.vm00.stdout:(65/150): librabbitmq-0.11.0-7.el9.x86_64.rpm 813 kB/s | 45 kB 00:00 2026-03-31T21:00:17.339 INFO:teuthology.orchestra.run.vm00.stdout:(66/150): zip-3.0-35.el9.x86_64.rpm 447 kB/s | 266 kB 00:00 2026-03-31T21:00:17.385 INFO:teuthology.orchestra.run.vm00.stdout:(67/150): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-03-31T21:00:17.389 INFO:teuthology.orchestra.run.vm00.stdout:(68/150): libstoragemgmt-1.10.1-1.el9.x86_64.rp 4.6 MB/s | 246 kB 00:00 2026-03-31T21:00:17.437 INFO:teuthology.orchestra.run.vm00.stdout:(69/150): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.4 MB/s | 292 kB 00:00 2026-03-31T21:00:17.441 INFO:teuthology.orchestra.run.vm00.stdout:(70/150): lua-5.4.4-4.el9.x86_64.rpm 3.6 MB/s | 188 kB 00:00 2026-03-31T21:00:17.488 INFO:teuthology.orchestra.run.vm00.stdout:(71/150): openblas-0.3.29-1.el9.x86_64.rpm 843 kB/s | 42 kB 00:00 2026-03-31T21:00:17.540 INFO:teuthology.orchestra.run.vm00.stdout:(72/150): perl-Benchmark-1.23-483.el9.noarch.rp 511 kB/s | 26 kB 00:00 2026-03-31T21:00:17.622 INFO:teuthology.orchestra.run.vm00.stdout:(73/150): libxslt-1.1.34-12.el9.x86_64.rpm 825 kB/s | 233 kB 00:00 2026-03-31T21:00:17.742 INFO:teuthology.orchestra.run.vm00.stdout:(74/150): openblas-openmp-0.3.29-1.el9.x86_64.r 18 MB/s | 5.3 MB 00:00 2026-03-31T21:00:17.768 INFO:teuthology.orchestra.run.vm00.stdout:(75/150): protobuf-3.14.0-17.el9.x86_64.rpm 6.9 MB/s | 1.0 MB 00:00 2026-03-31T21:00:17.820 INFO:teuthology.orchestra.run.vm00.stdout:(76/150): python3-devel-3.9.25-3.el9.x86_64.rpm 4.6 MB/s | 244 kB 00:00 2026-03-31T21:00:17.874 INFO:teuthology.orchestra.run.vm00.stdout:(77/150): python3-jinja2-2.11.3-8.el9.noarch.rp 4.5 MB/s | 249 kB 00:00 2026-03-31T21:00:17.924 INFO:teuthology.orchestra.run.vm00.stdout:(78/150): python3-jmespath-1.0.1-1.el9.noarch.r 954 kB/s | 48 kB 00:00 2026-03-31T21:00:17.944 INFO:teuthology.orchestra.run.vm00.stdout:(79/150): perl-Test-Harness-3.42-461.el9.noarch 730 kB/s | 295 kB 00:00 2026-03-31T21:00:17.977 INFO:teuthology.orchestra.run.vm00.stdout:(80/150): python3-libstoragemgmt-1.10.1-1.el9.x 3.3 MB/s | 177 kB 00:00 2026-03-31T21:00:18.028 INFO:teuthology.orchestra.run.vm00.stdout:(81/150): python3-markupsafe-1.1.1-12.el9.x86_6 682 kB/s | 35 kB 00:00 2026-03-31T21:00:18.180 INFO:teuthology.orchestra.run.vm00.stdout:(82/150): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2026-03-31T21:00:18.235 INFO:teuthology.orchestra.run.vm00.stdout:(83/150): python3-numpy-f2py-1.23.5-2.el9.x86_6 7.9 MB/s | 442 kB 00:00 2026-03-31T21:00:18.286 INFO:teuthology.orchestra.run.vm00.stdout:(84/150): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2026-03-31T21:00:18.339 INFO:teuthology.orchestra.run.vm00.stdout:(85/150): python3-protobuf-3.14.0-17.el9.noarch 4.9 MB/s | 267 kB 00:00 2026-03-31T21:00:18.401 INFO:teuthology.orchestra.run.vm00.stdout:(86/150): python3-numpy-1.23.5-2.el9.x86_64.rpm 16 MB/s | 6.1 MB 00:00 2026-03-31T21:00:18.402 INFO:teuthology.orchestra.run.vm00.stdout:(87/150): python3-pyasn1-0.4.8-7.el9.noarch.rpm 2.5 MB/s | 157 kB 00:00 2026-03-31T21:00:18.405 INFO:teuthology.orchestra.run.vm00.stdout:(88/150): python3-lxml-4.6.5-3.el9.x86_64.rpm 2.7 MB/s | 1.2 MB 00:00 2026-03-31T21:00:18.454 INFO:teuthology.orchestra.run.vm00.stdout:(89/150): python3-pyasn1-modules-0.4.8-7.el9.no 5.1 MB/s | 277 kB 00:00 2026-03-31T21:00:18.455 INFO:teuthology.orchestra.run.vm00.stdout:(90/150): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2026-03-31T21:00:18.504 INFO:teuthology.orchestra.run.vm00.stdout:(91/150): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2026-03-31T21:00:18.508 INFO:teuthology.orchestra.run.vm00.stdout:(92/150): qatlib-25.08.0-2.el9.x86_64.rpm 4.4 MB/s | 240 kB 00:00 2026-03-31T21:00:18.554 INFO:teuthology.orchestra.run.vm00.stdout:(93/150): qatlib-service-25.08.0-2.el9.x86_64.r 751 kB/s | 37 kB 00:00 2026-03-31T21:00:18.560 INFO:teuthology.orchestra.run.vm00.stdout:(94/150): qatzip-libs-1.3.1-1.el9.x86_64.rpm 1.3 MB/s | 66 kB 00:00 2026-03-31T21:00:18.610 INFO:teuthology.orchestra.run.vm00.stdout:(95/150): socat-1.7.4.1-8.el9.x86_64.rpm 5.4 MB/s | 303 kB 00:00 2026-03-31T21:00:18.611 INFO:teuthology.orchestra.run.vm00.stdout:(96/150): xmlsec1-1.2.29-13.el9.x86_64.rpm 3.6 MB/s | 189 kB 00:00 2026-03-31T21:00:18.661 INFO:teuthology.orchestra.run.vm00.stdout:(97/150): xmlsec1-openssl-1.2.29-13.el9.x86_64. 1.7 MB/s | 90 kB 00:00 2026-03-31T21:00:18.666 INFO:teuthology.orchestra.run.vm00.stdout:(98/150): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2026-03-31T21:00:18.944 INFO:teuthology.orchestra.run.vm00.stdout:(99/150): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2026-03-31T21:00:18.956 INFO:teuthology.orchestra.run.vm00.stdout:(100/150): abseil-cpp-20211102.0-4.el9.x86_64.r 43 MB/s | 551 kB 00:00 2026-03-31T21:00:18.963 INFO:teuthology.orchestra.run.vm00.stdout:(101/150): gperftools-libs-2.9.1-3.el9.x86_64.r 46 MB/s | 308 kB 00:00 2026-03-31T21:00:18.966 INFO:teuthology.orchestra.run.vm00.stdout:(102/150): grpc-data-1.46.7-10.el9.noarch.rpm 8.3 MB/s | 19 kB 00:00 2026-03-31T21:00:19.029 INFO:teuthology.orchestra.run.vm00.stdout:(103/150): libarrow-9.0.0-15.el9.x86_64.rpm 70 MB/s | 4.4 MB 00:00 2026-03-31T21:00:19.034 INFO:teuthology.orchestra.run.vm00.stdout:(104/150): libarrow-doc-9.0.0-15.el9.noarch.rpm 5.1 MB/s | 25 kB 00:00 2026-03-31T21:00:19.037 INFO:teuthology.orchestra.run.vm00.stdout:(105/150): liboath-2.6.12-1.el9.x86_64.rpm 17 MB/s | 49 kB 00:00 2026-03-31T21:00:19.041 INFO:teuthology.orchestra.run.vm00.stdout:(106/150): libunwind-1.6.2-1.el9.x86_64.rpm 19 MB/s | 67 kB 00:00 2026-03-31T21:00:19.045 INFO:teuthology.orchestra.run.vm00.stdout:(107/150): luarocks-3.9.2-5.el9.noarch.rpm 33 MB/s | 151 kB 00:00 2026-03-31T21:00:19.058 INFO:teuthology.orchestra.run.vm00.stdout:(108/150): parquet-libs-9.0.0-15.el9.x86_64.rpm 66 MB/s | 838 kB 00:00 2026-03-31T21:00:19.068 INFO:teuthology.orchestra.run.vm00.stdout:(109/150): python3-asyncssh-2.13.2-5.el9.noarch 55 MB/s | 548 kB 00:00 2026-03-31T21:00:19.071 INFO:teuthology.orchestra.run.vm00.stdout:(110/150): python3-autocommand-2.2.2-8.el9.noar 9.8 MB/s | 29 kB 00:00 2026-03-31T21:00:19.075 INFO:teuthology.orchestra.run.vm00.stdout:(111/150): python3-backports-tarfile-1.2.0-1.el 18 MB/s | 60 kB 00:00 2026-03-31T21:00:19.084 INFO:teuthology.orchestra.run.vm00.stdout:(112/150): python3-bcrypt-3.2.2-1.el9.x86_64.rp 5.1 MB/s | 43 kB 00:00 2026-03-31T21:00:19.086 INFO:teuthology.orchestra.run.vm00.stdout:(113/150): python3-cachetools-4.2.4-1.el9.noarc 13 MB/s | 32 kB 00:00 2026-03-31T21:00:19.089 INFO:teuthology.orchestra.run.vm00.stdout:(114/150): python3-certifi-2023.05.07-4.el9.noa 5.4 MB/s | 14 kB 00:00 2026-03-31T21:00:19.094 INFO:teuthology.orchestra.run.vm00.stdout:(115/150): python3-cheroot-10.0.1-5.el9.noarch. 36 MB/s | 173 kB 00:00 2026-03-31T21:00:19.100 INFO:teuthology.orchestra.run.vm00.stdout:(116/150): python3-cherrypy-18.10.0-5.el9.noarc 45 MB/s | 290 kB 00:00 2026-03-31T21:00:19.107 INFO:teuthology.orchestra.run.vm00.stdout:(117/150): python3-google-auth-2.45.0-1.el9.noa 42 MB/s | 254 kB 00:00 2026-03-31T21:00:19.141 INFO:teuthology.orchestra.run.vm00.stdout:(118/150): python3-grpcio-1.46.7-10.el9.x86_64. 60 MB/s | 2.0 MB 00:00 2026-03-31T21:00:19.145 INFO:teuthology.orchestra.run.vm00.stdout:(119/150): python3-grpcio-tools-1.46.7-10.el9.x 32 MB/s | 144 kB 00:00 2026-03-31T21:00:19.150 INFO:teuthology.orchestra.run.vm00.stdout:(120/150): python3-influxdb-5.3.1-1.el9.noarch. 30 MB/s | 139 kB 00:00 2026-03-31T21:00:19.154 INFO:teuthology.orchestra.run.vm00.stdout:(121/150): python3-isodate-0.6.1-3.el9.noarch.r 17 MB/s | 56 kB 00:00 2026-03-31T21:00:19.158 INFO:teuthology.orchestra.run.vm00.stdout:(122/150): python3-jaraco-8.2.1-3.el9.noarch.rp 2.5 MB/s | 11 kB 00:00 2026-03-31T21:00:19.161 INFO:teuthology.orchestra.run.vm00.stdout:(123/150): python3-jaraco-classes-3.2.1-5.el9.n 7.0 MB/s | 18 kB 00:00 2026-03-31T21:00:19.163 INFO:teuthology.orchestra.run.vm00.stdout:(124/150): python3-jaraco-collections-3.0.0-8.e 8.6 MB/s | 23 kB 00:00 2026-03-31T21:00:19.166 INFO:teuthology.orchestra.run.vm00.stdout:(125/150): python3-jaraco-context-6.0.1-3.el9.n 8.0 MB/s | 20 kB 00:00 2026-03-31T21:00:19.168 INFO:teuthology.orchestra.run.vm00.stdout:(126/150): python3-jaraco-functools-3.5.0-2.el9 8.0 MB/s | 19 kB 00:00 2026-03-31T21:00:19.175 INFO:teuthology.orchestra.run.vm00.stdout:(127/150): python3-jaraco-text-4.0.0-2.el9.noar 4.2 MB/s | 26 kB 00:00 2026-03-31T21:00:19.192 INFO:teuthology.orchestra.run.vm00.stdout:(128/150): python3-kubernetes-26.1.0-3.el9.noar 61 MB/s | 1.0 MB 00:00 2026-03-31T21:00:19.197 INFO:teuthology.orchestra.run.vm00.stdout:(129/150): python3-more-itertools-8.12.0-2.el9. 16 MB/s | 79 kB 00:00 2026-03-31T21:00:19.201 INFO:teuthology.orchestra.run.vm00.stdout:(130/150): python3-msgpack-1.0.3-2.el9.x86_64.r 23 MB/s | 86 kB 00:00 2026-03-31T21:00:19.207 INFO:teuthology.orchestra.run.vm00.stdout:(131/150): python3-natsort-7.1.1-5.el9.noarch.r 9.2 MB/s | 58 kB 00:00 2026-03-31T21:00:19.210 INFO:teuthology.orchestra.run.vm00.stdout:(132/150): python3-portend-3.1.0-2.el9.noarch.r 6.6 MB/s | 16 kB 00:00 2026-03-31T21:00:19.214 INFO:teuthology.orchestra.run.vm00.stdout:(133/150): python3-pyOpenSSL-21.0.0-1.el9.noarc 20 MB/s | 90 kB 00:00 2026-03-31T21:00:19.217 INFO:teuthology.orchestra.run.vm00.stdout:(134/150): python3-repoze-lru-0.7-16.el9.noarch 11 MB/s | 31 kB 00:00 2026-03-31T21:00:19.222 INFO:teuthology.orchestra.run.vm00.stdout:(135/150): python3-routes-2.5.1-5.el9.noarch.rp 36 MB/s | 188 kB 00:00 2026-03-31T21:00:19.226 INFO:teuthology.orchestra.run.vm00.stdout:(136/150): python3-rsa-4.9-2.el9.noarch.rpm 16 MB/s | 59 kB 00:00 2026-03-31T21:00:19.231 INFO:teuthology.orchestra.run.vm00.stdout:(137/150): python3-saml-1.16.0-1.el9.noarch.rpm 27 MB/s | 125 kB 00:00 2026-03-31T21:00:19.234 INFO:teuthology.orchestra.run.vm00.stdout:(138/150): python3-tempora-5.0.0-2.el9.noarch.r 13 MB/s | 36 kB 00:00 2026-03-31T21:00:19.237 INFO:teuthology.orchestra.run.vm00.stdout:(139/150): python3-typing-extensions-4.15.0-1.e 24 MB/s | 86 kB 00:00 2026-03-31T21:00:19.241 INFO:teuthology.orchestra.run.vm00.stdout:(140/150): python3-websocket-client-1.2.3-2.el9 23 MB/s | 90 kB 00:00 2026-03-31T21:00:19.245 INFO:teuthology.orchestra.run.vm00.stdout:(141/150): python3-xmlsec-1.3.13-1.el9.x86_64.r 15 MB/s | 48 kB 00:00 2026-03-31T21:00:19.247 INFO:teuthology.orchestra.run.vm00.stdout:(142/150): python3-xmltodict-0.12.0-15.el9.noar 8.5 MB/s | 22 kB 00:00 2026-03-31T21:00:19.250 INFO:teuthology.orchestra.run.vm00.stdout:(143/150): python3-zc-lockfile-2.0-10.el9.noarc 7.9 MB/s | 20 kB 00:00 2026-03-31T21:00:19.256 INFO:teuthology.orchestra.run.vm00.stdout:(144/150): re2-20211101-20.el9.x86_64.rpm 31 MB/s | 191 kB 00:00 2026-03-31T21:00:19.262 INFO:teuthology.orchestra.run.vm00.stdout:(145/150): s3cmd-2.4.0-1.el9.noarch.rpm 38 MB/s | 206 kB 00:00 2026-03-31T21:00:19.289 INFO:teuthology.orchestra.run.vm00.stdout:(146/150): thrift-0.15.0-4.el9.x86_64.rpm 58 MB/s | 1.6 MB 00:00 2026-03-31T21:00:19.445 INFO:teuthology.orchestra.run.vm00.stdout:(147/150): python3-scipy-1.9.3-2.el9.x86_64.rpm 19 MB/s | 19 MB 00:01 2026-03-31T21:00:19.514 INFO:teuthology.orchestra.run.vm00.stdout:(148/150): protobuf-compiler-3.14.0-17.el9.x86_ 1.0 MB/s | 862 kB 00:00 2026-03-31T21:00:20.372 INFO:teuthology.orchestra.run.vm00.stdout:(149/150): librados2-20.2.0-721.g5bb32787.el9.x 3.3 MB/s | 3.5 MB 00:01 2026-03-31T21:00:20.389 INFO:teuthology.orchestra.run.vm00.stdout:(150/150): librbd1-20.2.0-721.g5bb32787.el9.x86 3.0 MB/s | 2.8 MB 00:00 2026-03-31T21:00:20.394 INFO:teuthology.orchestra.run.vm00.stdout:-------------------------------------------------------------------------------- 2026-03-31T21:00:20.394 INFO:teuthology.orchestra.run.vm00.stdout:Total 19 MB/s | 270 MB 00:13 2026-03-31T21:00:21.172 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:00:21.249 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:00:21.249 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:00:22.325 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:00:22.325 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:00:23.480 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:00:23.489 INFO:teuthology.orchestra.run.vm00.stdout: Installing : thrift-0.15.0-4.el9.x86_64 1/152 2026-03-31T21:00:23.492 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/152 2026-03-31T21:00:23.504 INFO:teuthology.orchestra.run.vm00.stdout: Installing : liboath-2.6.12-1.el9.x86_64 3/152 2026-03-31T21:00:23.691 INFO:teuthology.orchestra.run.vm00.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/152 2026-03-31T21:00:23.693 INFO:teuthology.orchestra.run.vm00.stdout: Upgrading : librados2-2:20.2.0-721.g5bb32787.el9.x86_64 5/152 2026-03-31T21:00:23.751 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librados2-2:20.2.0-721.g5bb32787.el9.x86_64 5/152 2026-03-31T21:00:23.753 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 6/152 2026-03-31T21:00:23.770 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 6/152 2026-03-31T21:00:23.774 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libcephfs-daemon-2:20.2.0-721.g5bb32787.el9.x86_ 7/152 2026-03-31T21:00:23.775 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_ 8/152 2026-03-31T21:00:23.808 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_ 8/152 2026-03-31T21:00:23.816 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-rados-2:20.2.0-721.g5bb32787.el9.x86_64 9/152 2026-03-31T21:00:23.827 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 10/152 2026-03-31T21:00:23.834 INFO:teuthology.orchestra.run.vm00.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 11/152 2026-03-31T21:00:23.845 INFO:teuthology.orchestra.run.vm00.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 12/152 2026-03-31T21:00:23.849 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 13/152 2026-03-31T21:00:23.855 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 14/152 2026-03-31T21:00:24.008 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 15/152 2026-03-31T21:00:24.010 INFO:teuthology.orchestra.run.vm00.stdout: Upgrading : librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 16/152 2026-03-31T21:00:24.069 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 16/152 2026-03-31T21:00:24.078 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 17/152 2026-03-31T21:00:24.089 INFO:teuthology.orchestra.run.vm00.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 18/152 2026-03-31T21:00:24.090 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 19/152 2026-03-31T21:00:24.118 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 19/152 2026-03-31T21:00:24.120 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_ 20/152 2026-03-31T21:00:24.138 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_ 20/152 2026-03-31T21:00:24.175 INFO:teuthology.orchestra.run.vm00.stdout: Installing : re2-1:20211101-20.el9.x86_64 21/152 2026-03-31T21:00:24.201 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 22/152 2026-03-31T21:00:24.214 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 23/152 2026-03-31T21:00:24.222 INFO:teuthology.orchestra.run.vm00.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 24/152 2026-03-31T21:00:24.225 INFO:teuthology.orchestra.run.vm00.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/152 2026-03-31T21:00:24.234 INFO:teuthology.orchestra.run.vm00.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 26/152 2026-03-31T21:00:24.264 INFO:teuthology.orchestra.run.vm00.stdout: Installing : unzip-6.0-59.el9.x86_64 27/152 2026-03-31T21:00:24.282 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 28/152 2026-03-31T21:00:24.286 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 29/152 2026-03-31T21:00:24.293 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 30/152 2026-03-31T21:00:24.295 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 31/152 2026-03-31T21:00:24.337 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 32/152 2026-03-31T21:00:24.344 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-ceph-common-2:20.2.0-721.g5bb32787.el9.x 33/152 2026-03-31T21:00:24.354 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-ceph-argparse-2:20.2.0-721.g5bb32787.el9 34/152 2026-03-31T21:00:24.369 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-cephfs-2:20.2.0-721.g5bb32787.el9.x86_64 35/152 2026-03-31T21:00:24.377 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 36/152 2026-03-31T21:00:24.407 INFO:teuthology.orchestra.run.vm00.stdout: Installing : zip-3.0-35.el9.x86_64 37/152 2026-03-31T21:00:24.414 INFO:teuthology.orchestra.run.vm00.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/152 2026-03-31T21:00:24.424 INFO:teuthology.orchestra.run.vm00.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 39/152 2026-03-31T21:00:24.487 INFO:teuthology.orchestra.run.vm00.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 40/152 2026-03-31T21:00:24.505 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 41/152 2026-03-31T21:00:24.508 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/152 2026-03-31T21:00:24.517 INFO:teuthology.orchestra.run.vm00.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 43/152 2026-03-31T21:00:24.536 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 44/152 2026-03-31T21:00:24.544 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-rbd-2:20.2.0-721.g5bb32787.el9.x86_64 45/152 2026-03-31T21:00:24.550 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/152 2026-03-31T21:00:24.560 INFO:teuthology.orchestra.run.vm00.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 47/152 2026-03-31T21:00:24.569 INFO:teuthology.orchestra.run.vm00.stdout: Installing : librados-devel-2:20.2.0-721.g5bb32787.el9.x86_64 48/152 2026-03-31T21:00:24.575 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/152 2026-03-31T21:00:24.593 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 50/152 2026-03-31T21:00:24.600 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 51/152 2026-03-31T21:00:24.607 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 52/152 2026-03-31T21:00:24.625 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/152 2026-03-31T21:00:24.638 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/152 2026-03-31T21:00:24.648 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 55/152 2026-03-31T21:00:24.673 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 56/152 2026-03-31T21:00:24.688 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 57/152 2026-03-31T21:00:24.708 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 58/152 2026-03-31T21:00:24.715 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 59/152 2026-03-31T21:00:24.725 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 60/152 2026-03-31T21:00:24.774 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/152 2026-03-31T21:00:25.161 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 62/152 2026-03-31T21:00:25.177 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 63/152 2026-03-31T21:00:25.183 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 64/152 2026-03-31T21:00:25.190 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 65/152 2026-03-31T21:00:25.196 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 66/152 2026-03-31T21:00:25.204 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 67/152 2026-03-31T21:00:25.208 INFO:teuthology.orchestra.run.vm00.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 68/152 2026-03-31T21:00:25.210 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 69/152 2026-03-31T21:00:25.245 INFO:teuthology.orchestra.run.vm00.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 70/152 2026-03-31T21:00:25.299 INFO:teuthology.orchestra.run.vm00.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 71/152 2026-03-31T21:00:25.315 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 72/152 2026-03-31T21:00:25.325 INFO:teuthology.orchestra.run.vm00.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 73/152 2026-03-31T21:00:25.330 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/152 2026-03-31T21:00:25.337 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/152 2026-03-31T21:00:25.342 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 76/152 2026-03-31T21:00:25.351 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/152 2026-03-31T21:00:25.357 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/152 2026-03-31T21:00:25.392 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/152 2026-03-31T21:00:25.408 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 80/152 2026-03-31T21:00:25.417 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/152 2026-03-31T21:00:25.427 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 82/152 2026-03-31T21:00:25.471 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 83/152 2026-03-31T21:00:25.778 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 84/152 2026-03-31T21:00:25.819 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 85/152 2026-03-31T21:00:25.824 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 86/152 2026-03-31T21:00:25.830 INFO:teuthology.orchestra.run.vm00.stdout: Installing : perl-Benchmark-1.23-483.el9.noarch 87/152 2026-03-31T21:00:25.896 INFO:teuthology.orchestra.run.vm00.stdout: Installing : openblas-0.3.29-1.el9.x86_64 88/152 2026-03-31T21:00:25.899 INFO:teuthology.orchestra.run.vm00.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 89/152 2026-03-31T21:00:25.925 INFO:teuthology.orchestra.run.vm00.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 90/152 2026-03-31T21:00:26.320 INFO:teuthology.orchestra.run.vm00.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 91/152 2026-03-31T21:00:26.428 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 92/152 2026-03-31T21:00:27.294 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 93/152 2026-03-31T21:00:27.321 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 94/152 2026-03-31T21:00:27.330 INFO:teuthology.orchestra.run.vm00.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 95/152 2026-03-31T21:00:27.669 INFO:teuthology.orchestra.run.vm00.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 96/152 2026-03-31T21:00:27.673 INFO:teuthology.orchestra.run.vm00.stdout: Installing : librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 97/152 2026-03-31T21:00:27.697 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 97/152 2026-03-31T21:00:27.699 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-rgw-2:20.2.0-721.g5bb32787.el9.x86_64 98/152 2026-03-31T21:00:29.036 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 99/152 2026-03-31T21:00:29.041 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 99/152 2026-03-31T21:00:29.077 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 99/152 2026-03-31T21:00:29.080 INFO:teuthology.orchestra.run.vm00.stdout: Installing : smartmontools-1:7.2-10.el9.x86_64 100/152 2026-03-31T21:00:29.095 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: smartmontools-1:7.2-10.el9.x86_64 100/152 2026-03-31T21:00:29.095 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/smartd.service → /usr/lib/systemd/system/smartd.service. 2026-03-31T21:00:29.095 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:29.115 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 101/152 2026-03-31T21:00:29.125 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-packaging-20.9-5.el9.noarch 102/152 2026-03-31T21:00:29.145 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-ply-3.11-14.el9.noarch 103/152 2026-03-31T21:00:29.167 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 104/152 2026-03-31T21:00:29.271 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 105/152 2026-03-31T21:00:29.285 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 106/152 2026-03-31T21:00:29.323 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 107/152 2026-03-31T21:00:29.393 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-cheroot-10.0.1-5.el9.noarch 108/152 2026-03-31T21:00:29.453 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-cherrypy-18.10.0-5.el9.noarch 109/152 2026-03-31T21:00:29.463 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 110/152 2026-03-31T21:00:29.469 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 111/152 2026-03-31T21:00:29.475 INFO:teuthology.orchestra.run.vm00.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 112/152 2026-03-31T21:00:29.479 INFO:teuthology.orchestra.run.vm00.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 113/152 2026-03-31T21:00:29.481 INFO:teuthology.orchestra.run.vm00.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 114/152 2026-03-31T21:00:29.501 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 114/152 2026-03-31T21:00:29.634 INFO:teuthology.orchestra.run.vm00.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 115/152 2026-03-31T21:00:29.639 INFO:teuthology.orchestra.run.vm00.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 116/152 2026-03-31T21:00:29.944 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 116/152 2026-03-31T21:00:29.944 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-03-31T21:00:29.945 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:30.298 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 117/152 2026-03-31T21:00:30.360 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 117/152 2026-03-31T21:00:30.360 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-03-31T21:00:30.360 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-03-31T21:00:30.360 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:30.367 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 118/152 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 118/152 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /sys 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /proc 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /mnt 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /var/tmp 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /home 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /root 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /tmp 2026-03-31T21:00:36.708 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:36.842 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 119/152 2026-03-31T21:00:36.867 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 119/152 2026-03-31T21:00:36.867 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:36.867 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-03-31T21:00:36.867 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-03-31T21:00:36.867 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-03-31T21:00:36.867 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:37.131 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 120/152 2026-03-31T21:00:37.156 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 120/152 2026-03-31T21:00:37.156 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:37.156 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-03-31T21:00:37.156 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-03-31T21:00:37.156 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-03-31T21:00:37.156 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:37.166 INFO:teuthology.orchestra.run.vm00.stdout: Installing : mailcap-2.1.49-5.el9.noarch 121/152 2026-03-31T21:00:37.170 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 122/152 2026-03-31T21:00:37.192 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 123/152 2026-03-31T21:00:37.192 INFO:teuthology.orchestra.run.vm00.stdout:Creating group 'qat' with GID 994. 2026-03-31T21:00:37.192 INFO:teuthology.orchestra.run.vm00.stdout:Creating group 'libstoragemgmt' with GID 993. 2026-03-31T21:00:37.192 INFO:teuthology.orchestra.run.vm00.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 993 and GID 993. 2026-03-31T21:00:37.192 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:37.203 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 123/152 2026-03-31T21:00:37.237 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 123/152 2026-03-31T21:00:37.237 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-03-31T21:00:37.237 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:37.259 INFO:teuthology.orchestra.run.vm00.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 124/152 2026-03-31T21:00:37.287 INFO:teuthology.orchestra.run.vm00.stdout: Installing : fuse-2.9.9-17.el9.x86_64 125/152 2026-03-31T21:00:37.366 INFO:teuthology.orchestra.run.vm00.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 126/152 2026-03-31T21:00:37.370 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 127/152 2026-03-31T21:00:37.387 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 127/152 2026-03-31T21:00:37.387 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:37.387 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-03-31T21:00:37.387 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:38.221 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 128/152 2026-03-31T21:00:38.245 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 128/152 2026-03-31T21:00:38.245 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:38.245 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-03-31T21:00:38.245 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-03-31T21:00:38.245 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-03-31T21:00:38.245 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:38.313 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: cephadm-2:20.2.0-721.g5bb32787.el9.noarch 129/152 2026-03-31T21:00:38.317 INFO:teuthology.orchestra.run.vm00.stdout: Installing : cephadm-2:20.2.0-721.g5bb32787.el9.noarch 129/152 2026-03-31T21:00:38.326 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-prometheus-alerts-2:20.2.0-721.g5bb32787.el 130/152 2026-03-31T21:00:38.356 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-grafana-dashboards-2:20.2.0-721.g5bb32787.e 131/152 2026-03-31T21:00:38.359 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noar 132/152 2026-03-31T21:00:39.696 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noar 132/152 2026-03-31T21:00:39.707 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.no 133/152 2026-03-31T21:00:40.235 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.no 133/152 2026-03-31T21:00:40.237 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb3 134/152 2026-03-31T21:00:40.251 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb3 134/152 2026-03-31T21:00:40.253 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.no 135/152 2026-03-31T21:00:40.320 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.no 135/152 2026-03-31T21:00:40.373 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mgr-modules-core-2:20.2.0-721.g5bb32787.el9 136/152 2026-03-31T21:00:40.376 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 137/152 2026-03-31T21:00:40.402 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 137/152 2026-03-31T21:00:40.402 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:40.402 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-03-31T21:00:40.402 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-03-31T21:00:40.402 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-03-31T21:00:40.402 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:40.417 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 138/152 2026-03-31T21:00:40.431 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 138/152 2026-03-31T21:00:40.481 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-2:20.2.0-721.g5bb32787.el9.x86_64 139/152 2026-03-31T21:00:41.719 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 140/152 2026-03-31T21:00:41.722 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 141/152 2026-03-31T21:00:41.748 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 141/152 2026-03-31T21:00:41.748 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:41.748 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-03-31T21:00:41.748 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-03-31T21:00:41.748 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-03-31T21:00:41.748 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:41.761 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-immutable-object-cache-2:20.2.0-721.g5bb327 142/152 2026-03-31T21:00:41.785 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-721.g5bb327 142/152 2026-03-31T21:00:41.785 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:41.785 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-03-31T21:00:41.785 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:41.936 INFO:teuthology.orchestra.run.vm00.stdout: Installing : rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 143/152 2026-03-31T21:00:41.958 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 143/152 2026-03-31T21:00:41.958 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:00:41.958 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-03-31T21:00:41.959 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-03-31T21:00:41.959 INFO:teuthology.orchestra.run.vm00.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-03-31T21:00:41.959 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:46.415 INFO:teuthology.orchestra.run.vm00.stdout: Installing : ceph-test-2:20.2.0-721.g5bb32787.el9.x86_64 144/152 2026-03-31T21:00:46.423 INFO:teuthology.orchestra.run.vm00.stdout: Installing : perl-Test-Harness-1:3.42-461.el9.noarch 145/152 2026-03-31T21:00:46.430 INFO:teuthology.orchestra.run.vm00.stdout: Installing : libcephfs-devel-2:20.2.0-721.g5bb32787.el9.x86_6 146/152 2026-03-31T21:00:46.443 INFO:teuthology.orchestra.run.vm00.stdout: Installing : rbd-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 147/152 2026-03-31T21:00:46.462 INFO:teuthology.orchestra.run.vm00.stdout: Installing : rbd-nbd-2:20.2.0-721.g5bb32787.el9.x86_64 148/152 2026-03-31T21:00:46.470 INFO:teuthology.orchestra.run.vm00.stdout: Installing : s3cmd-2.4.0-1.el9.noarch 149/152 2026-03-31T21:00:46.474 INFO:teuthology.orchestra.run.vm00.stdout: Installing : bzip2-1.0.8-11.el9.x86_64 150/152 2026-03-31T21:00:46.474 INFO:teuthology.orchestra.run.vm00.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 151/152 2026-03-31T21:00:46.492 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 151/152 2026-03-31T21:00:46.493 INFO:teuthology.orchestra.run.vm00.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-2:20.2.0-721.g5bb32787.el9.x86_64 1/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 2/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 3/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 4/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-721.g5bb327 5/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 6/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 7/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 8/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 9/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 10/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 11/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-test-2:20.2.0-721.g5bb32787.el9.x86_64 12/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs-daemon-2:20.2.0-721.g5bb32787.el9.x86_ 13/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs-devel-2:20.2.0-721.g5bb32787.el9.x86_6 14/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_ 15/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 16/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 17/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librados-devel-2:20.2.0-721.g5bb32787.el9.x86_64 18/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_ 19/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 20/152 2026-03-31T21:00:47.961 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-ceph-argparse-2:20.2.0-721.g5bb32787.el9 21/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-ceph-common-2:20.2.0-721.g5bb32787.el9.x 22/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cephfs-2:20.2.0-721.g5bb32787.el9.x86_64 23/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rados-2:20.2.0-721.g5bb32787.el9.x86_64 24/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rbd-2:20.2.0-721.g5bb32787.el9.x86_64 25/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rgw-2:20.2.0-721.g5bb32787.el9.x86_64 26/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : rbd-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 27/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 28/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : rbd-nbd-2:20.2.0-721.g5bb32787.el9.x86_64 29/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-721.g5bb32787.e 30/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noar 31/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.no 32/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb3 33/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.no 34/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-721.g5bb32787.el9 35/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 36/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-721.g5bb32787.el 37/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 38/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : cephadm-2:20.2.0-721.g5bb32787.el9.noarch 39/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : bzip2-1.0.8-11.el9.x86_64 40/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 41/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : fuse-2.9.9-17.el9.x86_64 42/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 43/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 44/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 45/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 46/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 47/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 48/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 49/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 50/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 51/152 2026-03-31T21:00:47.962 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-ply-3.11-14.el9.noarch 52/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 53/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 54/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 55/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 56/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : smartmontools-1:7.2-10.el9.x86_64 57/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : unzip-6.0-59.el9.x86_64 58/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : zip-3.0-35.el9.x86_64 59/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 60/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 61/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 62/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 63/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 64/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 65/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 66/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 67/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 68/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 69/152 2026-03-31T21:00:47.963 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 70/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : lua-5.4.4-4.el9.x86_64 71/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 72/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 73/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : perl-Benchmark-1.23-483.el9.noarch 74/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : perl-Test-Harness-1:3.42-461.el9.noarch 75/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 76/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 78/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 79/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 80/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 81/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 82/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 84/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 85/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 87/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 88/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 89/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 90/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 91/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 92/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 93/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 94/152 2026-03-31T21:00:47.964 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 95/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 96/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 97/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 98/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 99/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 100/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 101/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 102/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 103/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 104/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 105/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 106/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 107/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 108/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 109/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 110/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 111/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 112/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 113/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 114/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 115/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 116/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cheroot-10.0.1-5.el9.noarch 117/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cherrypy-18.10.0-5.el9.noarch 118/152 2026-03-31T21:00:47.965 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 119/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 120/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 121/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 122/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 123/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 124/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 125/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 126/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 127/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 129/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 130/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 131/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 132/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 134/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 135/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 136/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 137/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 138/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 139/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 140/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 141/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 142/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 143/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 144/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 145/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : re2-1:20211101-20.el9.x86_64 146/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : s3cmd-2.4.0-1.el9.noarch 147/152 2026-03-31T21:00:47.966 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 148/152 2026-03-31T21:00:47.967 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librados2-2:20.2.0-721.g5bb32787.el9.x86_64 149/152 2026-03-31T21:00:47.967 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/152 2026-03-31T21:00:47.967 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 151/152 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/152 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout:Upgraded: 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: librados2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout:Installed: 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: bzip2-1.0.8-11.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-grafana-dashboards-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-immutable-object-cache-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-modules-core-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-prometheus-alerts-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-test-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: cephadm-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: fuse-2.9.9-17.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: grpc-data-1.46.7-10.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: libarrow-9.0.0-15.el9.x86_64 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-03-31T21:00:48.080 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-daemon-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-devel-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libconfig-1.7.2-9.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libnbd-1.20.3-4.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: liboath-2.6.12-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: librados-devel-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libunwind-1.6.2-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: libxslt-1.1.34-12.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: lua-5.4.4-4.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: luarocks-3.9.2-5.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: mailcap-2.1.49-5.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: nvme-cli-2.16-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: openblas-0.3.29-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: pciutils-3.7.0-7.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: perl-Benchmark-1.23-483.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: perl-Test-Harness-1:3.42-461.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: protobuf-3.14.0-17.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-babel-2.9.1-2.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-argparse-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-cephfs-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-cheroot-10.0.1-5.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-cherrypy-18.10.0-5.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-03-31T21:00:48.081 INFO:teuthology.orchestra.run.vm00.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-packaging-20.9-5.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-ply-3.11-14.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-portend-3.1.0-2.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-pycparser-2.20-6.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-rados-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-rbd-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests-2.25.1-10.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-rgw-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-routes-2.5.1-5.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-rsa-4.9-2.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-saml-1.16.0-1.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-toml-0.10.2-6.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: qatlib-25.08.0-2.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: rbd-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: rbd-nbd-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: re2-1:20211101-20.el9.x86_64 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: s3cmd-2.4.0-1.el9.noarch 2026-03-31T21:00:48.082 INFO:teuthology.orchestra.run.vm00.stdout: smartmontools-1:7.2-10.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: socat-1.7.4.1-8.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: thrift-0.15.0-4.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: unzip-6.0-59.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: zip-3.0-35.el9.x86_64 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:48.083 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:00:48.173 DEBUG:teuthology.parallel:result is None 2026-03-31T21:00:48.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5bb3278730741031382ca9c3dc9d221a942e06a2 2026-03-31T21:00:48.777 DEBUG:teuthology.orchestra.run.vm00:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-03-31T21:00:48.797 INFO:teuthology.orchestra.run.vm00.stdout:20.2.0-721.g5bb32787.el9 2026-03-31T21:00:48.797 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-721.g5bb32787.el9 2026-03-31T21:00:48.797 INFO:teuthology.task.install:The correct ceph version 20.2.0-721.g5bb32787 is installed. 2026-03-31T21:00:48.797 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-03-31T21:00:48.797 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:48.797 DEBUG:teuthology.orchestra.run.vm00:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-03-31T21:00:48.864 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-03-31T21:00:48.864 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:48.864 DEBUG:teuthology.orchestra.run.vm00:> sudo dd of=/usr/bin/daemon-helper 2026-03-31T21:00:48.931 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-03-31T21:00:49.000 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-03-31T21:00:49.000 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:49.000 DEBUG:teuthology.orchestra.run.vm00:> sudo dd of=/usr/bin/adjust-ulimits 2026-03-31T21:00:49.063 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-03-31T21:00:49.128 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-03-31T21:00:49.128 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:49.128 DEBUG:teuthology.orchestra.run.vm00:> sudo dd of=/usr/bin/stdin-killer 2026-03-31T21:00:49.193 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-03-31T21:00:49.258 INFO:teuthology.run_tasks:Running task ceph... 2026-03-31T21:00:49.298 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-03-31T21:00:49.298 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod 777 /var/log/ceph 2026-03-31T21:00:49.322 INFO:tasks.ceph:Disabling ceph logrotate... 2026-03-31T21:00:49.322 DEBUG:teuthology.orchestra.run.vm00:> sudo rm -f -- /etc/logrotate.d/ceph 2026-03-31T21:00:49.388 INFO:tasks.ceph:Creating extra log directories... 2026-03-31T21:00:49.388 DEBUG:teuthology.orchestra.run.vm00:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-03-31T21:00:49.456 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-03-31T21:00:49.456 INFO:tasks.ceph:config {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'avl', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '256K', 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock iops capacity threshold hdd': 49000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SLOW_OPS\\)', 'deep-scrub', 'missing', 'overall HEALTH_', 'repair', 'slow request', 'unfound', '\\(POOL_APP_NOT_ENABLED\\)', 'enough copies available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-03-31T21:00:49.456 INFO:tasks.ceph:ctx.config {'archive_path': '/archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348', 'branch': 'tentacle', 'description': 'rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{avl} base mem$/{normal-2} onode-segment$/{256K} write$/{random/{compr$/{no$/{no}} random}}}} rados supported-random-distro$/{centos_latest}}', 'email': None, 'first_in_suite': False, 'flavor': 'default', 'job_id': '4348', 'last_in_suite': False, 'machine_type': 'vps', 'name': 'kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'tentacle'}, 'ansible.cephlab': {'branch': 'main', 'repo': 'https://github.com/kshtsk/ceph-cm-ansible.git', 'skip_tags': 'nagios,monitoring-scripts,hostname,pubkeys,zap,sudoers,kerberos,ntp-client,resolvconf,cpan,nfs', 'vars': {'logical_volumes': {'lv_1': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_2': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_3': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}, 'lv_4': {'scratch_dev': True, 'size': '25%VG', 'vg': 'vg_nvme'}}, 'timezone': 'UTC', 'volume_groups': {'vg_nvme': {'pvs': '/dev/vdb,/dev/vdc,/dev/vdd,/dev/vde'}}}}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'avl', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '256K', 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock iops capacity threshold hdd': 49000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '5bb3278730741031382ca9c3dc9d221a942e06a2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_binary_url': 'https://download.ceph.com/rpm-20.2.0/el9/noarch/cephadm'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5bb3278730741031382ca9c3dc9d221a942e06a2'}, 'extra_system_packages': {'deb': ['python3-jmespath', 'python3-xmltodict', 's3cmd'], 'rpm': ['bzip2', 'perl-Test-Harness', 'python3-jmespath', 'python3-xmltodict', 's3cmd']}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'tt-tentacle', 'sha1': '0392f78529848ec72469e8e431875cb98d3a5fb4'}}, 'owner': 'kyr', 'priority': 1000, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 6407, 'sha1': '5bb3278730741031382ca9c3dc9d221a942e06a2', 'sleep_before_teardown': 0, 'subset': '1/100000', 'suite': 'rados', 'suite_branch': 'tt-tentacle', 'suite_path': '/home/teuthos/src/github.com_kshtsk_ceph_0392f78529848ec72469e8e431875cb98d3a5fb4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/kshtsk/ceph.git', 'suite_sha1': '0392f78529848ec72469e8e431875cb98d3a5fb4', 'targets': {'vm00.local': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAnNB2WMelB5Hnf3uHsc711hbb+KoMw3iMoXVqaJL51aDJu0TdvEFEuQE8CKQEmUxO8zosMwhwGoucJq8WTttG8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'avl', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '256K', 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock iops capacity threshold hdd': 49000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'create_rbd_pool': False, 'log-ignorelist': ['\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SLOW_OPS\\)', 'deep-scrub', 'missing', 'overall HEALTH_', 'repair', 'slow request', 'unfound', '\\(POOL_APP_NOT_ENABLED\\)', 'enough copies available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': '5bb3278730741031382ca9c3dc9d221a942e06a2', 'cluster': 'ceph'}}, {'ec_inconsistent_hinfo': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'uv2', 'teuthology_repo': 'https://github.com/kshtsk/teuthology', 'teuthology_sha1': 'a59626679648f962bca99d20d35578f2998c8f37', 'timestamp': '2026-03-31_11:18:10', 'tube': 'vps', 'user': 'kyr', 'verbose': False, 'worker_log': '/home/teuthos/.teuthology/dispatcher/dispatcher.vps.282426'} 2026-03-31T21:00:49.456 DEBUG:teuthology.orchestra.run.vm00:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-03-31T21:00:49.508 DEBUG:teuthology.orchestra.run.vm00:> sudo install -d -m0777 -- /var/run/ceph 2026-03-31T21:00:49.572 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:49.572 DEBUG:teuthology.orchestra.run.vm00:> dd if=/scratch_devs of=/dev/stdout 2026-03-31T21:00:49.626 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-03-31T21:00:49.627 DEBUG:teuthology.orchestra.run.vm00:> stat /dev/vg_nvme/lv_1 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout:Device: 6h/6d Inode: 968 Links: 1 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout:Context: system_u:object_r:device_t:s0 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout:Access: 2026-03-31 21:00:46.823852331 +0000 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout:Modify: 2026-03-31 21:00:30.020825128 +0000 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout:Change: 2026-03-31 21:00:30.020825128 +0000 2026-03-31T21:00:49.683 INFO:teuthology.orchestra.run.vm00.stdout: Birth: 2026-03-31 21:00:30.020825128 +0000 2026-03-31T21:00:49.683 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-03-31T21:00:49.746 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records in 2026-03-31T21:00:49.747 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records out 2026-03-31T21:00:49.747 INFO:teuthology.orchestra.run.vm00.stderr:512 bytes copied, 0.000209371 s, 2.4 MB/s 2026-03-31T21:00:49.748 DEBUG:teuthology.orchestra.run.vm00:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-03-31T21:00:49.805 DEBUG:teuthology.orchestra.run.vm00:> stat /dev/vg_nvme/lv_2 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout:Device: 6h/6d Inode: 957 Links: 1 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout:Context: system_u:object_r:device_t:s0 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout:Access: 2026-03-31 21:00:46.823852331 +0000 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout:Modify: 2026-03-31 21:00:30.018825124 +0000 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout:Change: 2026-03-31 21:00:30.018825124 +0000 2026-03-31T21:00:49.863 INFO:teuthology.orchestra.run.vm00.stdout: Birth: 2026-03-31 21:00:30.018825124 +0000 2026-03-31T21:00:49.863 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-03-31T21:00:49.927 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records in 2026-03-31T21:00:49.927 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records out 2026-03-31T21:00:49.927 INFO:teuthology.orchestra.run.vm00.stderr:512 bytes copied, 0.000169146 s, 3.0 MB/s 2026-03-31T21:00:49.929 DEBUG:teuthology.orchestra.run.vm00:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-03-31T21:00:49.986 DEBUG:teuthology.orchestra.run.vm00:> stat /dev/vg_nvme/lv_3 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout:Device: 6h/6d Inode: 955 Links: 1 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout:Context: system_u:object_r:device_t:s0 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout:Access: 2026-03-31 21:00:46.824852333 +0000 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout:Modify: 2026-03-31 21:00:30.018825124 +0000 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout:Change: 2026-03-31 21:00:30.018825124 +0000 2026-03-31T21:00:50.042 INFO:teuthology.orchestra.run.vm00.stdout: Birth: 2026-03-31 21:00:30.018825124 +0000 2026-03-31T21:00:50.042 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-03-31T21:00:50.108 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records in 2026-03-31T21:00:50.108 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records out 2026-03-31T21:00:50.108 INFO:teuthology.orchestra.run.vm00.stderr:512 bytes copied, 0.000140593 s, 3.6 MB/s 2026-03-31T21:00:50.109 DEBUG:teuthology.orchestra.run.vm00:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-03-31T21:00:50.166 DEBUG:teuthology.orchestra.run.vm00:> stat /dev/vg_nvme/lv_4 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout:Device: 6h/6d Inode: 973 Links: 1 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout:Context: system_u:object_r:device_t:s0 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout:Access: 2026-03-31 21:00:46.824852333 +0000 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout:Modify: 2026-03-31 21:00:30.025825136 +0000 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout:Change: 2026-03-31 21:00:30.025825136 +0000 2026-03-31T21:00:50.222 INFO:teuthology.orchestra.run.vm00.stdout: Birth: 2026-03-31 21:00:30.025825136 +0000 2026-03-31T21:00:50.223 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-03-31T21:00:50.285 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records in 2026-03-31T21:00:50.285 INFO:teuthology.orchestra.run.vm00.stderr:1+0 records out 2026-03-31T21:00:50.285 INFO:teuthology.orchestra.run.vm00.stderr:512 bytes copied, 0.000132177 s, 3.9 MB/s 2026-03-31T21:00:50.286 DEBUG:teuthology.orchestra.run.vm00:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-03-31T21:00:50.345 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-31T21:00:50.345 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@vm00.local'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-03-31T21:00:50.345 INFO:tasks.ceph:Generating config... 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd max pg log entries = 5 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bdev async discard = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bluestore allocator = avl 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bluestore onode segment size = 256K 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] debug ms = 1 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] debug osd = 20 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd mclock iops capacity threshold hdd = 49000 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd mclock skip benchmark = True 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-03-31T21:00:50.346 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-03-31T21:00:50.347 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] mon election default strategy = 3 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] ms bind msgr1 = False 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] ms bind msgr2 = True 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2026-03-31T21:00:50.347 INFO:tasks.ceph:[global] ms type = async 2026-03-31T21:00:50.347 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-03-31T21:00:50.347 INFO:tasks.ceph:[mgr] debug monc = 10 2026-03-31T21:00:50.347 INFO:tasks.ceph:[mgr] debug ms = 1 2026-03-31T21:00:50.347 INFO:tasks.ceph:[mon] debug mon = 20 2026-03-31T21:00:50.347 INFO:tasks.ceph:[mon] debug ms = 1 2026-03-31T21:00:50.347 INFO:tasks.ceph:[mon] debug paxos = 20 2026-03-31T21:00:50.347 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-03-31T21:00:50.347 INFO:tasks.ceph:Setting up mon.a... 2026-03-31T21:00:50.347 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-03-31T21:00:50.421 INFO:teuthology.orchestra.run.vm00.stdout:creating /etc/ceph/ceph.keyring 2026-03-31T21:00:50.423 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-03-31T21:00:50.502 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-03-31T21:00:50.567 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '192.168.123.100'), ('mon.b', '[v2:192.168.123.100:3301,v1:192.168.123.100:6790]'), ('mon.c', '[v2:192.168.123.100:3302,v1:192.168.123.100:6791]')] 2026-03-31T21:00:50.568 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '192.168.123.100,[v2:192.168.123.100:3301,v1:192.168.123.100:6790],[v2:192.168.123.100:3302,v1:192.168.123.100:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': True, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'avl', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore onode segment size': '256K', 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock iops capacity threshold hdd': 49000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-03-31T21:00:50.568 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:50.568 DEBUG:teuthology.orchestra.run.vm00:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-03-31T21:00:50.624 DEBUG:teuthology.orchestra.run.vm00:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 192.168.123.100 --addv b '[v2:192.168.123.100:3301,v1:192.168.123.100:6790]' --addv c '[v2:192.168.123.100:3302,v1:192.168.123.100:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:monmaptool: generated fsid edf0be71-aa1c-40ef-819b-fbd038c1604f 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:setting min_mon_release = tentacle 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:epoch 0 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:fsid edf0be71-aa1c-40ef-819b-fbd038c1604f 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:last_changed 2026-03-31T21:00:50.699774+0000 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:created 2026-03-31T21:00:50.699774+0000 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:min_mon_release 20 (tentacle) 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:election_strategy: 3 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:0: [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] mon.a 2026-03-31T21:00:50.701 INFO:teuthology.orchestra.run.vm00.stdout:1: [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] mon.b 2026-03-31T21:00:50.702 INFO:teuthology.orchestra.run.vm00.stdout:2: [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] mon.c 2026-03-31T21:00:50.702 INFO:teuthology.orchestra.run.vm00.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-03-31T21:00:50.703 DEBUG:teuthology.orchestra.run.vm00:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-03-31T21:00:50.759 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID edf0be71-aa1c-40ef-819b-fbd038c1604f... 2026-03-31T21:00:50.759 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout:[global] 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: chdir = "" 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: auth supported = cephx 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: filestore xattr use omap = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon clock drift allowed = 1.000 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: osd crush chooseleaf type = 0 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: auth debug = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: ms die on old message = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: ms die on bug = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon max pg per osd = 10000 # >= luminous 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon pg warn max object skew = 0 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: # disable pg_autoscaler by default for new pools 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: osd_pool_default_pg_autoscale_mode = off 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: osd pool default size = 2 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon osd allow primary affinity = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon osd allow pg remap = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon warn on legacy crush tunables = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon warn on crush straw calc version zero = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon warn on no sortbitwise = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon warn on osd down out interval zero = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon warn on too few osds = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon_warn_on_pool_no_redundancy = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon_allow_pool_size_one = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: osd default data pool replay window = 5 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon allow pool delete = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon cluster log file level = debug 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: debug asserts on shutdown = true 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon health detail to clog = false 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon host = "192.168.123.100,[v2:192.168.123.100:3301,v1:192.168.123.100:6790],[v2:192.168.123.100:3302,v1:192.168.123.100:6791]" 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon client directed command retry = 5 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon client hunt interval max multiple = 2 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon election default strategy = 3 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: mon mgr beacon grace = 90 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: ms bind msgr1 = False 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: ms bind msgr2 = True 2026-03-31T21:00:50.843 INFO:teuthology.orchestra.run.vm00.stdout: ms inject socket failures = 1000 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: ms type = async 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: fsid = edf0be71-aa1c-40ef-819b-fbd038c1604f 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout:[osd] 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd journal size = 100 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd scrub load threshold = 5.0 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd scrub max interval = 600 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd mclock profile = high_recovery_ops 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd mclock skip benchmark = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd recover clone overlap = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd recovery max chunk = 1048576 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug shutdown = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug op order = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug verify stray on activate = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug trim objects = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd open classes on start = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug pg log writeout = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd deep scrub update digest min age = 30 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd map max advance = 10 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: journal zero on create = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: filestore ondisk finisher threads = 3 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: filestore apply finisher threads = 3 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bdev debug aio = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug misdirected ops = true 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd max pg log entries = 5 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd min pg log entries = 5 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bdev async discard = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bdev enable discard = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bluestore allocator = avl 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bluestore block size = 96636764160 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bluestore fsck on mount = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bluestore onode segment size = 256K 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bluestore write v2 random = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: bluestore zero block detection = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug bluefs = 20 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug bluestore = 20 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug ms = 1 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug osd = 20 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug rocksdb = 10 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: mon osd backfillfull_ratio = 0.85 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: mon osd full ratio = 0.9 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: mon osd nearfull ratio = 0.8 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug verify cached snaps = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd debug verify missing on start = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd failsafe full ratio = 0.95 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd mclock iops capacity threshold hdd = 49000 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd mclock override recovery settings = True 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd objectstore = bluestore 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd op queue = debug_random 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: osd op queue cut off = debug_random 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout:[mgr] 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug ms = 1 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug mgr = 20 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug mon = 20 2026-03-31T21:00:50.844 INFO:teuthology.orchestra.run.vm00.stdout: debug auth = 20 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon reweight min pgs per osd = 4 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon reweight min bytes per osd = 10 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mgr/telemetry/nag = false 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: debug monc = 10 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout:[mon] 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: debug ms = 1 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: debug mon = 20 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: debug paxos = 20 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: debug auth = 20 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon data avail warn = 5 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon mgr mkfs grace = 240 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon reweight min pgs per osd = 4 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon osd reporter subtree level = osd 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon osd prime pg temp = true 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon reweight min bytes per osd = 10 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: auth mon ticket ttl = 660 # 11m 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: auth service ticket ttl = 240 # 4m 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: # don't complain about insecure global_id in the test suite 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: # 1m isn't quite enough 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon_down_mkfs_grace = 2m 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon_warn_on_filestore_osds = false 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: mon scrub interval = 300 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout:[client] 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: rgw cache enabled = true 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: rgw enable ops log = true 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: rgw enable usage log = true 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout:[mon.a] 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout:[mon.b] 2026-03-31T21:00:50.845 INFO:teuthology.orchestra.run.vm00.stdout:[mon.c] 2026-03-31T21:00:50.854 INFO:tasks.ceph:Creating admin key on mon.a... 2026-03-31T21:00:50.854 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-03-31T21:00:50.942 INFO:tasks.ceph:Copying monmap to all nodes... 2026-03-31T21:00:50.942 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:50.942 DEBUG:teuthology.orchestra.run.vm00:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-03-31T21:00:50.956 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:50.957 DEBUG:teuthology.orchestra.run.vm00:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-03-31T21:00:51.013 INFO:tasks.ceph:Sending monmap to node ubuntu@vm00.local 2026-03-31T21:00:51.013 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:51.013 DEBUG:teuthology.orchestra.run.vm00:> sudo dd of=/etc/ceph/ceph.keyring 2026-03-31T21:00:51.013 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-03-31T21:00:51.090 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:51.090 DEBUG:teuthology.orchestra.run.vm00:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:51.148 INFO:tasks.ceph:Setting up mon nodes... 2026-03-31T21:00:51.148 INFO:tasks.ceph:Setting up mgr nodes... 2026-03-31T21:00:51.148 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-03-31T21:00:51.235 INFO:teuthology.orchestra.run.vm00.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-03-31T21:00:51.238 INFO:tasks.ceph:Setting up mds nodes... 2026-03-31T21:00:51.238 INFO:tasks.ceph_client:Setting up client nodes... 2026-03-31T21:00:51.238 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-03-31T21:00:51.238 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@vm00.local'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-03-31T21:00:51.238 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-03-31T21:00:51.262 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-31T21:00:51.262 INFO:tasks.ceph:role: osd.0 2026-03-31T21:00:51.262 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@vm00.local 2026-03-31T21:00:51.262 DEBUG:teuthology.orchestra.run.vm00:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=1310464 blks 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout: = sunit=0 swidth=0 blks 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-31T21:00:51.331 INFO:teuthology.orchestra.run.vm00.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-31T21:00:51.335 INFO:teuthology.orchestra.run.vm00.stdout:Discarding blocks...Done. 2026-03-31T21:00:51.346 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@vm00.local -o noatime 2026-03-31T21:00:51.346 DEBUG:teuthology.orchestra.run.vm00:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-03-31T21:00:51.422 DEBUG:teuthology.orchestra.run.vm00:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-03-31T21:00:51.490 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-03-31T21:00:51.555 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-31T21:00:51.556 INFO:tasks.ceph:role: osd.1 2026-03-31T21:00:51.556 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@vm00.local 2026-03-31T21:00:51.556 DEBUG:teuthology.orchestra.run.vm00:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=1310464 blks 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout: = sunit=0 swidth=0 blks 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-31T21:00:51.621 INFO:teuthology.orchestra.run.vm00.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-31T21:00:51.626 INFO:teuthology.orchestra.run.vm00.stdout:Discarding blocks...Done. 2026-03-31T21:00:51.629 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@vm00.local -o noatime 2026-03-31T21:00:51.629 DEBUG:teuthology.orchestra.run.vm00:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-03-31T21:00:51.700 DEBUG:teuthology.orchestra.run.vm00:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-03-31T21:00:51.768 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-03-31T21:00:51.833 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-31T21:00:51.834 INFO:tasks.ceph:role: osd.2 2026-03-31T21:00:51.834 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@vm00.local 2026-03-31T21:00:51.834 DEBUG:teuthology.orchestra.run.vm00:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=1310464 blks 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout: = sunit=0 swidth=0 blks 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-31T21:00:51.898 INFO:teuthology.orchestra.run.vm00.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-31T21:00:51.899 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-31T21:00:51.899 INFO:teuthology.orchestra.run.vm00.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-31T21:00:51.903 INFO:teuthology.orchestra.run.vm00.stdout:Discarding blocks...Done. 2026-03-31T21:00:51.906 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@vm00.local -o noatime 2026-03-31T21:00:51.906 DEBUG:teuthology.orchestra.run.vm00:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-03-31T21:00:51.979 DEBUG:teuthology.orchestra.run.vm00:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-03-31T21:00:52.048 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-03-31T21:00:52.114 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-03-31T21:00:52.114 INFO:tasks.ceph:role: osd.3 2026-03-31T21:00:52.114 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@vm00.local 2026-03-31T21:00:52.114 DEBUG:teuthology.orchestra.run.vm00:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-03-31T21:00:52.179 INFO:teuthology.orchestra.run.vm00.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=1310464 blks 2026-03-31T21:00:52.179 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 attr=2, projid32bit=1 2026-03-31T21:00:52.179 INFO:teuthology.orchestra.run.vm00.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-03-31T21:00:52.179 INFO:teuthology.orchestra.run.vm00.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-03-31T21:00:52.180 INFO:teuthology.orchestra.run.vm00.stdout:data = bsize=4096 blocks=5241856, imaxpct=25 2026-03-31T21:00:52.180 INFO:teuthology.orchestra.run.vm00.stdout: = sunit=0 swidth=0 blks 2026-03-31T21:00:52.180 INFO:teuthology.orchestra.run.vm00.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-03-31T21:00:52.180 INFO:teuthology.orchestra.run.vm00.stdout:log =internal log bsize=4096 blocks=16384, version=2 2026-03-31T21:00:52.180 INFO:teuthology.orchestra.run.vm00.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2026-03-31T21:00:52.180 INFO:teuthology.orchestra.run.vm00.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-03-31T21:00:52.185 INFO:teuthology.orchestra.run.vm00.stdout:Discarding blocks...Done. 2026-03-31T21:00:52.187 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@vm00.local -o noatime 2026-03-31T21:00:52.187 DEBUG:teuthology.orchestra.run.vm00:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-03-31T21:00:52.257 DEBUG:teuthology.orchestra.run.vm00:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-03-31T21:00:52.324 DEBUG:teuthology.orchestra.run.vm00:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:52.408 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:52.405+0000 7fee78725900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-03-31T21:00:52.408 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:52.405+0000 7fee78725900 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-03-31T21:00:52.408 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:52.405+0000 7fee78725900 -1 bdev(0x55ea1a4dd800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-03-31T21:00:52.408 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:52.405+0000 7fee78725900 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-03-31T21:00:53.033 DEBUG:teuthology.orchestra.run.vm00:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-03-31T21:00:53.059 DEBUG:teuthology.orchestra.run.vm00:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:53.139 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.136+0000 7f78aca13900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-03-31T21:00:53.139 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.136+0000 7f78aca13900 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-03-31T21:00:53.139 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.136+0000 7f78aca13900 -1 bdev(0x55bf0ee0f800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-03-31T21:00:53.139 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.136+0000 7f78aca13900 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-03-31T21:00:53.773 DEBUG:teuthology.orchestra.run.vm00:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-03-31T21:00:53.839 DEBUG:teuthology.orchestra.run.vm00:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:53.922 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.919+0000 7fc8701f4900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-03-31T21:00:53.922 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.919+0000 7fc8701f4900 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-03-31T21:00:53.922 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.919+0000 7fc8701f4900 -1 bdev(0x55ed07583800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-03-31T21:00:53.922 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:53.920+0000 7fc8701f4900 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-03-31T21:00:54.575 DEBUG:teuthology.orchestra.run.vm00:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-03-31T21:00:54.643 DEBUG:teuthology.orchestra.run.vm00:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:54.723 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:54.720+0000 7fe845811900 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-03-31T21:00:54.723 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:54.720+0000 7fe845811900 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-03-31T21:00:54.723 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:54.720+0000 7fe845811900 -1 bdev(0x55a5c8009800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-03-31T21:00:54.723 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:00:54.720+0000 7fe845811900 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-03-31T21:00:55.382 DEBUG:teuthology.orchestra.run.vm00:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-03-31T21:00:55.452 INFO:tasks.ceph:Reading keys from all nodes... 2026-03-31T21:00:55.452 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:55.452 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-03-31T21:00:55.516 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:55.516 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-03-31T21:00:55.582 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:55.582 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-03-31T21:00:55.645 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:55.645 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-03-31T21:00:55.713 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:55.713 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-03-31T21:00:55.778 INFO:tasks.ceph:Adding keys to all mons... 2026-03-31T21:00:55.778 DEBUG:teuthology.orchestra.run.vm00:> sudo tee -a /etc/ceph/ceph.keyring 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout:[mgr.x] 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout: key = AQADNsxp6P/tDRAA4GeWSTHoaZltHV24qvuSPw== 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout:[osd.0] 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout: key = AQAENsxpW6o9GBAAJP/td7dqEo4z2lWJvt9KTg== 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout:[osd.1] 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout: key = AQAFNsxpi+AxCBAApyWK9IMVgkvoQPEPWEblFA== 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout:[osd.2] 2026-03-31T21:00:55.841 INFO:teuthology.orchestra.run.vm00.stdout: key = AQAFNsxpTP7gNhAA/tZ7CrG99RQihOryZPoyKQ== 2026-03-31T21:00:55.842 INFO:teuthology.orchestra.run.vm00.stdout:[osd.3] 2026-03-31T21:00:55.842 INFO:teuthology.orchestra.run.vm00.stdout: key = AQAGNsxpCGMDKxAAwasvDPPRCUfY2SxnTrAfPQ== 2026-03-31T21:00:55.842 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-03-31T21:00:55.927 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-31T21:00:55.971 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-31T21:00:56.016 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-31T21:00:56.060 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-03-31T21:00:56.103 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-03-31T21:00:56.103 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-03-31T21:00:56.166 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-03-31T21:00:56.265 DEBUG:teuthology.orchestra.run.vm00:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-03-31T21:00:56.291 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-03-31T21:00:56.358 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-03-31T21:00:56.452 DEBUG:teuthology.orchestra.run.vm00:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-03-31T21:00:56.478 DEBUG:teuthology.orchestra.run.vm00:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-03-31T21:00:56.542 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-03-31T21:00:56.636 DEBUG:teuthology.orchestra.run.vm00:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-03-31T21:00:56.661 DEBUG:teuthology.orchestra.run.vm00:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-03-31T21:00:56.717 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-03-31T21:00:56.717 INFO:tasks.ceph.mon.a:Restarting daemon 2026-03-31T21:00:56.717 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-03-31T21:00:56.760 INFO:tasks.ceph.mon.a:Started 2026-03-31T21:00:56.761 INFO:tasks.ceph.mon.b:Restarting daemon 2026-03-31T21:00:56.761 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-03-31T21:00:56.762 INFO:tasks.ceph.mon.b:Started 2026-03-31T21:00:56.762 INFO:tasks.ceph.mon.c:Restarting daemon 2026-03-31T21:00:56.762 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-03-31T21:00:56.764 INFO:tasks.ceph.mon.c:Started 2026-03-31T21:00:56.765 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2026-03-31T21:00:57.110 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-03-31T21:00:57.110 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-03-31T21:00:57.110 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-03-31T21:00:57.112 INFO:tasks.ceph.mgr.x:Started 2026-03-31T21:00:57.112 DEBUG:tasks.ceph:set 0 configs 2026-03-31T21:00:57.112 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph --cluster ceph config dump 2026-03-31T21:00:57.230 INFO:teuthology.orchestra.run.vm00.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-03-31T21:00:57.230 INFO:teuthology.orchestra.run.vm00.stdout:mgr unknown mgr/devicehealth/enable_monitoring false * 2026-03-31T21:00:57.243 INFO:tasks.ceph:Setting crush tunables to default 2026-03-31T21:00:57.243 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph --cluster ceph osd crush tunables default 2026-03-31T21:00:57.369 INFO:teuthology.orchestra.run.vm00.stderr:adjusted tunables profile to default 2026-03-31T21:00:57.388 INFO:tasks.ceph:check_enable_crimson: False 2026-03-31T21:00:57.388 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-03-31T21:00:57.388 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:57.388 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-03-31T21:00:57.419 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:57.419 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-03-31T21:00:57.487 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:57.487 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-03-31T21:00:57.554 DEBUG:teuthology.orchestra.run.vm00:> set -ex 2026-03-31T21:00:57.554 DEBUG:teuthology.orchestra.run.vm00:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-03-31T21:00:57.623 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph --cluster ceph osd new 403a5788-99d7-42ee-96ed-c2891fdebaf1 0 2026-03-31T21:00:57.793 INFO:teuthology.orchestra.run.vm00.stdout:0 2026-03-31T21:00:57.805 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph --cluster ceph osd new c1dccf3e-c41f-46a2-8fbd-09a0651739c6 1 2026-03-31T21:00:57.902 INFO:tasks.ceph.mgr.x.vm00.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-03-31T21:00:57.902 INFO:tasks.ceph.mgr.x.vm00.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-03-31T21:00:57.902 INFO:tasks.ceph.mgr.x.vm00.stderr: from numpy import show_config as show_numpy_config 2026-03-31T21:00:57.941 INFO:teuthology.orchestra.run.vm00.stdout:1 2026-03-31T21:00:57.957 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph --cluster ceph osd new f9e6dead-91ad-448b-b46d-36fdf906cfa7 2 2026-03-31T21:00:58.090 INFO:teuthology.orchestra.run.vm00.stdout:2 2026-03-31T21:00:58.106 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph --cluster ceph osd new 3b7f3900-5638-4954-97cf-d86d7e1e2462 3 2026-03-31T21:00:58.246 INFO:teuthology.orchestra.run.vm00.stdout:3 2026-03-31T21:00:58.257 INFO:tasks.ceph.osd.0:Restarting daemon 2026-03-31T21:00:58.257 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-03-31T21:00:58.259 INFO:tasks.ceph.osd.0:Started 2026-03-31T21:00:58.259 INFO:tasks.ceph.osd.1:Restarting daemon 2026-03-31T21:00:58.259 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-03-31T21:00:58.260 INFO:tasks.ceph.osd.1:Started 2026-03-31T21:00:58.260 INFO:tasks.ceph.osd.2:Restarting daemon 2026-03-31T21:00:58.260 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-03-31T21:00:58.261 INFO:tasks.ceph.osd.2:Started 2026-03-31T21:00:58.261 INFO:tasks.ceph.osd.3:Restarting daemon 2026-03-31T21:00:58.262 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-03-31T21:00:58.267 INFO:tasks.ceph.osd.3:Started 2026-03-31T21:00:58.267 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:00:58.445 INFO:tasks.ceph.osd.1.vm00.stderr:2026-03-31T21:00:58.442+0000 7f5452259900 -1 Falling back to public interface 2026-03-31T21:00:58.449 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:00:58.446+0000 7f5185872900 -1 Falling back to public interface 2026-03-31T21:00:58.449 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:00:58.449 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":6,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:00:58.239148+0000","last_up_change":"0.000000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:00:58.461 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-03-31T21:00:58.461 INFO:tasks.ceph:Waiting for OSDs to come up 2026-03-31T21:00:58.465 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:00:58.462+0000 7f81647a7900 -1 Falling back to public interface 2026-03-31T21:00:58.466 INFO:tasks.ceph.osd.0.vm00.stderr:2026-03-31T21:00:58.463+0000 7fb80f419900 -1 Falling back to public interface 2026-03-31T21:00:58.863 DEBUG:teuthology.orchestra.run.vm00:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-03-31T21:00:58.896 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:00:58.893+0000 7f5185872900 -1 osd.3 0 log_to_monitors true 2026-03-31T21:00:58.943 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:00:58.940+0000 7f81647a7900 -1 osd.2 0 log_to_monitors true 2026-03-31T21:00:58.947 INFO:tasks.ceph.osd.1.vm00.stderr:2026-03-31T21:00:58.944+0000 7f5452259900 -1 osd.1 0 log_to_monitors true 2026-03-31T21:00:58.964 INFO:tasks.ceph.osd.0.vm00.stderr:2026-03-31T21:00:58.960+0000 7fb80f419900 -1 osd.0 0 log_to_monitors true 2026-03-31T21:00:58.999 INFO:teuthology.misc.health.vm00.stdout: 2026-03-31T21:00:58.999 INFO:teuthology.misc.health.vm00.stdout:{"epoch":6,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:00:58.239148+0000","last_up_change":"0.000000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:00:59.008 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-03-31T21:01:00.794 INFO:tasks.ceph.mgr.x.vm00.stderr:2026-03-31T21:01:00.791+0000 7f8b82e9f640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-03-31T21:01:00.795 INFO:tasks.ceph.mgr.x.vm00.stderr:2026-03-31T21:01:00.792+0000 7f8b82e9f640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-03-31T21:01:01.070 INFO:tasks.ceph.osd.1.vm00.stderr:2026-03-31T21:01:01.067+0000 7f544d9c5640 -1 osd.1 0 waiting for initial osdmap 2026-03-31T21:01:01.072 INFO:tasks.ceph.osd.0.vm00.stderr:2026-03-31T21:01:01.069+0000 7fb80a380640 -1 osd.0 0 waiting for initial osdmap 2026-03-31T21:01:01.072 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:01.069+0000 7f5180fe0640 -1 osd.3 0 waiting for initial osdmap 2026-03-31T21:01:01.073 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:01:01.070+0000 7f815ff13640 -1 osd.2 0 waiting for initial osdmap 2026-03-31T21:01:01.076 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:01.072+0000 7f517cdc2640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:01:01.078 INFO:tasks.ceph.osd.1.vm00.stderr:2026-03-31T21:01:01.074+0000 7f54497a7640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:01:01.079 INFO:tasks.ceph.osd.0.vm00.stderr:2026-03-31T21:01:01.075+0000 7fb806162640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:01:01.080 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:01:01.077+0000 7f815b4f4640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:01:05.410 DEBUG:teuthology.orchestra.run.vm00:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-03-31T21:01:05.610 INFO:teuthology.misc.health.vm00.stdout: 2026-03-31T21:01:05.610 INFO:teuthology.misc.health.vm00.stdout:{"epoch":11,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:05.295513+0000","last_up_change":"2026-03-31T21:01:02.077235+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":1012853978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":1012853978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":1012853978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":1012853978}]},"public_addr":"192.168.123.100:6800/1012853978","cluster_addr":"192.168.123.100:6801/1012853978","heartbeat_back_addr":"192.168.123.100:6803/1012853978","heartbeat_front_addr":"192.168.123.100:6802/1012853978","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:01:05.621 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-03-31T21:01:05.621 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-03-31T21:01:05.621 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-03-31T21:01:05.621 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-03-31T21:01:05.843 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:05.854 INFO:teuthology.orchestra.run.vm00.stdout:[{"version":2,"timestamp":"2026-03-31T21:00:57.090644+0000","name":"","changes":[{"name":"mgr/mgr/devicehealth/enable_monitoring","new_value":"false"}]}] 2026-03-31T21:01:05.855 INFO:tasks.ceph_manager:config epoch is 2 2026-03-31T21:01:05.855 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-03-31T21:01:05.855 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-03-31T21:01:05.855 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-03-31T21:01:06.120 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:06.133 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6816","nonce":2853644495}]},"active_addr":"192.168.123.100:6816/2853644495","active_change":"2026-03-31T21:00:59.777197+0000","active_mgr_features":4541880224203014143,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"192.168.123.100:0","nonce":2371719305}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"192.168.123.100:0","nonce":1682565401}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"192.168.123.100:0","nonce":4139914409}]}]} 2026-03-31T21:01:06.134 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-03-31T21:01:06.134 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-03-31T21:01:06.134 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:01:06.415 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:06.412+0000 7f76bd480640 0 --2- 192.168.123.100:0/3188779642 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f76b8131ba0 0x7f76b8077410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:01:06.553 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:06.553 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":12,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:06.302399+0000","last_up_change":"2026-03-31T21:01:02.077235+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":1012853978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":1012853978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":1012853978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":1012853978}]},"public_addr":"192.168.123.100:6800/1012853978","cluster_addr":"192.168.123.100:6801/1012853978","heartbeat_back_addr":"192.168.123.100:6803/1012853978","heartbeat_front_addr":"192.168.123.100:6802/1012853978","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:01:06.565 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-03-31T21:01:06.565 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:01:06.775 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:06.775 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":12,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:06.302399+0000","last_up_change":"2026-03-31T21:01:02.077235+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":1012853978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":1012853978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":1012853978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":1012853978}]},"public_addr":"192.168.123.100:6800/1012853978","cluster_addr":"192.168.123.100:6801/1012853978","heartbeat_back_addr":"192.168.123.100:6803/1012853978","heartbeat_front_addr":"192.168.123.100:6802/1012853978","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:01:06.788 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:01:06.788 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:01:06.788 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:01:06.788 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:01:06.938 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:06.938 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:06.938 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:06.939 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:06.944 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:06.944 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:06.955 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:06.955 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:07.191 INFO:teuthology.orchestra.run.vm00.stdout:38654705665 2026-03-31T21:01:07.208 INFO:teuthology.orchestra.run.vm00.stdout:38654705665 2026-03-31T21:01:07.214 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.1 2026-03-31T21:01:07.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.3 2026-03-31T21:01:07.269 INFO:teuthology.orchestra.run.vm00.stdout:38654705665 2026-03-31T21:01:07.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.2 2026-03-31T21:01:07.304 INFO:teuthology.orchestra.run.vm00.stdout:38654705665 2026-03-31T21:01:07.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.0 2026-03-31T21:01:08.214 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:08.231 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:08.285 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:08.319 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:08.470 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:08.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.1 2026-03-31T21:01:08.492 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:08.532 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:08.551 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:08.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.2 2026-03-31T21:01:08.552 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:08.568 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.3 2026-03-31T21:01:08.568 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:08.624 INFO:teuthology.orchestra.run.vm00.stdout:38654705666 2026-03-31T21:01:08.638 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.0 2026-03-31T21:01:08.638 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:08.638 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-03-31T21:01:08.638 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:08.830 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:08.830 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:08.843 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-03-31T21:01:07.789609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108368,"kb_used_data":1040,"kb_used_omap":22,"kb_used_meta":107241,"kb_avail":377378992,"statfs":{"total":386547056640,"available":386436087808,"internally_reserved":0,"allocated":1064960,"data_stored":748020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22731,"internal_metadata":109815605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":36,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-03-31T21:01:06.458367+0000","last_change":"2026-03-31T21:01:05.306443+0000","last_active":"2026-03-31T21:01:06.458367+0000","last_peered":"2026-03-31T21:01:06.458367+0000","last_clean":"2026-03-31T21:01:06.458367+0000","last_became_active":"2026-03-31T21:01:05.306191+0000","last_became_peered":"2026-03-31T21:01:05.306191+0000","last_unstale":"2026-03-31T21:01:06.458367+0000","last_undegraded":"2026-03-31T21:01:06.458367+0000","last_fullsized":"2026-03-31T21:01:06.458367+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:36:03.018303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26928,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26810,"kb_avail":94344912,"statfs":{"total":96636764160,"available":96609189888,"internally_reserved":0,"allocated":98304,"data_stored":22154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27454065},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26928,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26810,"kb_avail":94344912,"statfs":{"total":96636764160,"available":96609189888,"internally_reserved":0,"allocated":98304,"data_stored":22154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5521,"internal_metadata":27454063},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":420,"kb_used_omap":5,"kb_used_meta":26810,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":430080,"data_stored":350346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5521,"internal_metadata":27454063},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":428,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":438272,"data_stored":353366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6170,"internal_metadata":27453414},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-31T21:01:08.843 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:09.037 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:09.037 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:09.049 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2026-03-31T21:01:07.789609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":108368,"kb_used_data":1040,"kb_used_omap":22,"kb_used_meta":107241,"kb_avail":377378992,"statfs":{"total":386547056640,"available":386436087808,"internally_reserved":0,"allocated":1064960,"data_stored":748020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22731,"internal_metadata":109815605},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"11'17","reported_seq":36,"reported_epoch":12,"state":"active+clean","last_fresh":"2026-03-31T21:01:06.458367+0000","last_change":"2026-03-31T21:01:05.306443+0000","last_active":"2026-03-31T21:01:06.458367+0000","last_peered":"2026-03-31T21:01:06.458367+0000","last_clean":"2026-03-31T21:01:06.458367+0000","last_became_active":"2026-03-31T21:01:05.306191+0000","last_became_peered":"2026-03-31T21:01:05.306191+0000","last_unstale":"2026-03-31T21:01:06.458367+0000","last_undegraded":"2026-03-31T21:01:06.458367+0000","last_fullsized":"2026-03-31T21:01:06.458367+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:36:03.018303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26928,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26810,"kb_avail":94344912,"statfs":{"total":96636764160,"available":96609189888,"internally_reserved":0,"allocated":98304,"data_stored":22154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5519,"internal_metadata":27454065},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26928,"kb_used_data":96,"kb_used_omap":5,"kb_used_meta":26810,"kb_avail":94344912,"statfs":{"total":96636764160,"available":96609189888,"internally_reserved":0,"allocated":98304,"data_stored":22154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5521,"internal_metadata":27454063},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":420,"kb_used_omap":5,"kb_used_meta":26810,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":430080,"data_stored":350346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5521,"internal_metadata":27454063},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705666,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":428,"kb_used_omap":6,"kb_used_meta":26809,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":438272,"data_stored":353366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6170,"internal_metadata":27453414},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-31T21:01:09.049 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-03-31T21:01:09.049 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-03-31T21:01:09.049 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-03-31T21:01:09.049 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-03-31T21:01:09.273 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:09.273 INFO:teuthology.orchestra.run.vm00.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-03-31T21:01:09.285 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-03-31T21:01:09.285 INFO:teuthology.run_tasks:Running task ec_inconsistent_hinfo... 2026-03-31T21:01:09.289 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:01:09.489 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:09.489 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":12,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:06.302399+0000","last_up_change":"2026-03-31T21:01:02.077235+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":1012853978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":1012853978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":1012853978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":1012853978}]},"public_addr":"192.168.123.100:6800/1012853978","cluster_addr":"192.168.123.100:6801/1012853978","heartbeat_back_addr":"192.168.123.100:6803/1012853978","heartbeat_front_addr":"192.168.123.100:6802/1012853978","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:01:09.500 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2026-03-31T21:01:03.806955+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'nonprimary_shards': '{}', 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-03-31T21:01:09.500 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2026-03-31T21:01:09.704 INFO:teuthology.orchestra.run.vm00.stdout:pg_num: 1 2026-03-31T21:01:09.717 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set backfill_unfound k=2 m=1 crush-failure-domain=osd 2026-03-31T21:01:10.343 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:creating pool_name unique_pool_0 2026-03-31T21:01:10.343 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 1 1 erasure backfill_unfound 2026-03-31T21:01:12.339 INFO:teuthology.orchestra.run.vm00.stderr:pool 'unique_pool_0' created 2026-03-31T21:01:12.357 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2026-03-31T21:01:13.346 INFO:teuthology.orchestra.run.vm00.stderr:set pool 2 min_size to 2 2026-03-31T21:01:13.361 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-03-31T21:01:14.359 INFO:teuthology.orchestra.run.vm00.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-03-31T21:01:15.372 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_autoscale_mode off 2026-03-31T21:01:16.468 INFO:teuthology.orchestra.run.vm00.stderr:set pool 2 pg_autoscale_mode to off 2026-03-31T21:01:16.483 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:01:16.483 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:01:16.483 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:01:16.484 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:01:16.597 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:16.597 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:16.609 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:16.610 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:16.611 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:16.611 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:16.620 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:16.620 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:16.838 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:16.854 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:16.854 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:16.863 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.3 2026-03-31T21:01:16.867 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.1 2026-03-31T21:01:16.869 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2026-03-31T21:01:16.887 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:16.900 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.2 2026-03-31T21:01:17.864 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:17.867 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:17.870 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:17.900 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:18.097 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:18.111 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.3 2026-03-31T21:01:18.115 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:18.128 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2026-03-31T21:01:18.128 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:18.140 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.2 2026-03-31T21:01:18.201 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:18.198+0000 7efeeffff640 0 --2- 192.168.123.100:0/1408218148 >> v2:192.168.123.100:3300/0 conn(0x7efee8017f20 0x7efee80d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:01:18.323 INFO:teuthology.orchestra.run.vm00.stdout:38654705668 2026-03-31T21:01:18.335 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705668 for osd.1 2026-03-31T21:01:19.112 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:19.128 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:19.141 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:19.324 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:19.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:19.336 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705669 for osd.3 2026-03-31T21:01:19.336 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:19.359 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:19.365 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:19.377 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705669 for osd.0 2026-03-31T21:01:19.377 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:19.378 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705669 for osd.2 2026-03-31T21:01:19.378 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:19.560 INFO:teuthology.orchestra.run.vm00.stdout:38654705669 2026-03-31T21:01:19.573 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705669 got 38654705669 for osd.1 2026-03-31T21:01:19.573 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:19.573 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-03-31T21:01:19.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:19.770 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:19.770 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:19.782 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2026-03-31T21:01:17.791020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108544,"kb_used_data":1216,"kb_used_omap":36,"kb_used_meta":107227,"kb_avail":377378816,"statfs":{"total":386547056640,"available":386435907584,"internally_reserved":0,"allocated":1245184,"data_stored":825641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37056,"internal_metadata":109801280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.452092"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.470016+0000","last_change":"2026-03-31T21:01:14.012630+0000","last_active":"2026-03-31T21:01:16.470016+0000","last_peered":"2026-03-31T21:01:16.470016+0000","last_clean":"2026-03-31T21:01:16.470016+0000","last_became_active":"2026-03-31T21:01:14.012445+0000","last_became_peered":"2026-03-31T21:01:14.012445+0000","last_unstale":"2026-03-31T21:01:16.470016+0000","last_undegraded":"2026-03-31T21:01:16.470016+0000","last_fullsized":"2026-03-31T21:01:16.470016+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_clean_scrub_stamp":"2026-03-31T21:01:12.331997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T21:58:24.675401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.468907+0000","last_change":"2026-03-31T21:01:05.306443+0000","last_active":"2026-03-31T21:01:16.468907+0000","last_peered":"2026-03-31T21:01:16.468907+0000","last_clean":"2026-03-31T21:01:16.468907+0000","last_became_active":"2026-03-31T21:01:05.306191+0000","last_became_peered":"2026-03-31T21:01:05.306191+0000","last_unstale":"2026-03-31T21:01:16.468907+0000","last_undegraded":"2026-03-31T21:01:16.468907+0000","last_fullsized":"2026-03-31T21:01:16.468907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:36:03.018303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26976,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344864,"statfs":{"total":96636764160,"available":96609140736,"internally_reserved":0,"allocated":147456,"data_stored":43246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9425,"internal_metadata":27450159},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26976,"kb_used_data":144,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344864,"statfs":{"total":96636764160,"available":96609140736,"internally_reserved":0,"allocated":147456,"data_stored":43246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9427,"internal_metadata":27450157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":468,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":479232,"data_stored":371438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9427,"internal_metadata":27450157},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":460,"kb_used_omap":8,"kb_used_meta":26807,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":471040,"data_stored":367711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8777,"internal_metadata":27450807},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-31T21:01:19.782 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:19.981 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:19.982 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:19.993 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-03-31T21:01:19.791301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108584,"kb_used_data":1256,"kb_used_omap":38,"kb_used_meta":107225,"kb_avail":377378776,"statfs":{"total":386547056640,"available":386435866624,"internally_reserved":0,"allocated":1286144,"data_stored":844276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39656,"internal_metadata":109798680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.450316"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.470016+0000","last_change":"2026-03-31T21:01:14.012630+0000","last_active":"2026-03-31T21:01:16.470016+0000","last_peered":"2026-03-31T21:01:16.470016+0000","last_clean":"2026-03-31T21:01:16.470016+0000","last_became_active":"2026-03-31T21:01:14.012445+0000","last_became_peered":"2026-03-31T21:01:14.012445+0000","last_unstale":"2026-03-31T21:01:16.470016+0000","last_undegraded":"2026-03-31T21:01:16.470016+0000","last_fullsized":"2026-03-31T21:01:16.470016+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_clean_scrub_stamp":"2026-03-31T21:01:12.331997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T21:58:24.675401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.468907+0000","last_change":"2026-03-31T21:01:05.306443+0000","last_active":"2026-03-31T21:01:16.468907+0000","last_peered":"2026-03-31T21:01:16.468907+0000","last_clean":"2026-03-31T21:01:16.468907+0000","last_became_active":"2026-03-31T21:01:05.306191+0000","last_became_peered":"2026-03-31T21:01:05.306191+0000","last_unstale":"2026-03-31T21:01:16.468907+0000","last_undegraded":"2026-03-31T21:01:16.468907+0000","last_fullsized":"2026-03-31T21:01:16.468907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:36:03.018303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":46973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27449509},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":46973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":487424,"data_stored":375165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":487424,"data_stored":375165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9427,"internal_metadata":27450157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-31T21:01:19.993 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-03-31T21:01:19.993 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:01:20.188 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:20.188 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":18,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:16.460993+0000","last_up_change":"2026-03-31T21:01:02.077235+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":1012853978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":1012853978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":1012853978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":1012853978}]},"public_addr":"192.168.123.100:6800/1012853978","cluster_addr":"192.168.123.100:6801/1012853978","heartbeat_back_addr":"192.168.123.100:6803/1012853978","heartbeat_front_addr":"192.168.123.100:6802/1012853978","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:01:20.200 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:20.390 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:20.390 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:20.402 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-03-31T21:01:19.791301+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108584,"kb_used_data":1256,"kb_used_omap":38,"kb_used_meta":107225,"kb_avail":377378776,"statfs":{"total":386547056640,"available":386435866624,"internally_reserved":0,"allocated":1286144,"data_stored":844276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39656,"internal_metadata":109798680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.450316"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.470016+0000","last_change":"2026-03-31T21:01:14.012630+0000","last_active":"2026-03-31T21:01:16.470016+0000","last_peered":"2026-03-31T21:01:16.470016+0000","last_clean":"2026-03-31T21:01:16.470016+0000","last_became_active":"2026-03-31T21:01:14.012445+0000","last_became_peered":"2026-03-31T21:01:14.012445+0000","last_unstale":"2026-03-31T21:01:16.470016+0000","last_undegraded":"2026-03-31T21:01:16.470016+0000","last_fullsized":"2026-03-31T21:01:16.470016+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_clean_scrub_stamp":"2026-03-31T21:01:12.331997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T21:58:24.675401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.468907+0000","last_change":"2026-03-31T21:01:05.306443+0000","last_active":"2026-03-31T21:01:16.468907+0000","last_peered":"2026-03-31T21:01:16.468907+0000","last_clean":"2026-03-31T21:01:16.468907+0000","last_became_active":"2026-03-31T21:01:05.306191+0000","last_became_peered":"2026-03-31T21:01:05.306191+0000","last_unstale":"2026-03-31T21:01:16.468907+0000","last_undegraded":"2026-03-31T21:01:16.468907+0000","last_fullsized":"2026-03-31T21:01:16.468907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:36:03.018303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":46973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27449509},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":46973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":487424,"data_stored":375165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":487424,"data_stored":375165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9427,"internal_metadata":27450157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-31T21:01:20.403 INFO:tasks.ec_inconsistent_hinfo:acting=[3, 1, 2] 2026-03-31T21:01:20.403 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2026-03-31T21:01:20.403 DEBUG:teuthology.orchestra.run.vm00:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2026-03-31T21:01:20.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:01:20.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:01:20.551 INFO:teuthology.orchestra.run.vm00.stdout:38654705671 2026-03-31T21:01:20.551 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:20.551 INFO:teuthology.orchestra.run.vm00.stdout:38654705671 2026-03-31T21:01:20.551 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:20.790 INFO:teuthology.orchestra.run.vm00.stdout:38654705670 2026-03-31T21:01:20.799 INFO:teuthology.orchestra.run.vm00.stdout:38654705670 2026-03-31T21:01:20.804 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705670 for osd.0 2026-03-31T21:01:20.813 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2026-03-31T21:01:21.805 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:21.813 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:22.018 INFO:teuthology.orchestra.run.vm00.stdout:38654705670 2026-03-31T21:01:22.030 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705670 for osd.0 2026-03-31T21:01:22.051 INFO:teuthology.orchestra.run.vm00.stdout:38654705670 2026-03-31T21:01:22.062 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2026-03-31T21:01:23.031 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:23.063 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:23.231 INFO:teuthology.orchestra.run.vm00.stdout:38654705671 2026-03-31T21:01:23.243 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705671 for osd.0 2026-03-31T21:01:23.244 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:23.274 INFO:teuthology.orchestra.run.vm00.stdout:38654705671 2026-03-31T21:01:23.286 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705671 for osd.1 2026-03-31T21:01:23.286 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:23.286 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for recovery to complete 2026-03-31T21:01:23.286 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:23.608 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:23.608 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:23.620 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-03-31T21:01:21.791589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108584,"kb_used_data":1256,"kb_used_omap":38,"kb_used_meta":107225,"kb_avail":377378776,"statfs":{"total":386547056640,"available":386435866624,"internally_reserved":0,"allocated":1286144,"data_stored":844276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39656,"internal_metadata":109798680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001087"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.470016+0000","last_change":"2026-03-31T21:01:14.012630+0000","last_active":"2026-03-31T21:01:16.470016+0000","last_peered":"2026-03-31T21:01:16.470016+0000","last_clean":"2026-03-31T21:01:16.470016+0000","last_became_active":"2026-03-31T21:01:14.012445+0000","last_became_peered":"2026-03-31T21:01:14.012445+0000","last_unstale":"2026-03-31T21:01:16.470016+0000","last_undegraded":"2026-03-31T21:01:16.470016+0000","last_fullsized":"2026-03-31T21:01:16.470016+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_clean_scrub_stamp":"2026-03-31T21:01:12.331997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T21:58:24.675401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.468907+0000","last_change":"2026-03-31T21:01:05.306443+0000","last_active":"2026-03-31T21:01:16.468907+0000","last_peered":"2026-03-31T21:01:16.468907+0000","last_clean":"2026-03-31T21:01:16.468907+0000","last_became_active":"2026-03-31T21:01:05.306191+0000","last_became_peered":"2026-03-31T21:01:05.306191+0000","last_unstale":"2026-03-31T21:01:16.468907+0000","last_undegraded":"2026-03-31T21:01:16.468907+0000","last_fullsized":"2026-03-31T21:01:16.468907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:36:03.018303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":46973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27449509},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":46973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":487424,"data_stored":375165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":487424,"data_stored":375165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9427,"internal_metadata":27450157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-31T21:01:23.621 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:23.833 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:23.833 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:23.845 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2026-03-31T21:01:23.791814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328870,"num_objects":3,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":38,"num_write_kb":323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108596,"kb_used_data":1268,"kb_used_omap":38,"kb_used_meta":107225,"kb_avail":377378764,"statfs":{"total":386547056640,"available":386435854336,"internally_reserved":0,"allocated":1298432,"data_stored":856564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39656,"internal_metadata":109798680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":678,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.438582"},"pg_stats":[{"pgid":"2.0","version":"18'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.470076+0000","last_change":"2026-03-31T21:01:14.012630+0000","last_active":"2026-03-31T21:01:16.470076+0000","last_peered":"2026-03-31T21:01:16.470076+0000","last_clean":"2026-03-31T21:01:16.470076+0000","last_became_active":"2026-03-31T21:01:14.012445+0000","last_became_peered":"2026-03-31T21:01:14.012445+0000","last_unstale":"2026-03-31T21:01:16.470076+0000","last_undegraded":"2026-03-31T21:01:16.470076+0000","last_fullsized":"2026-03-31T21:01:16.470076+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:12.331997+0000","last_clean_scrub_stamp":"2026-03-31T21:01:12.331997+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T21:58:24.675401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":678,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":48,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-03-31T21:01:16.468907+0000","last_change":"2026-03-31T21:01:05.306443+0000","last_active":"2026-03-31T21:01:16.468907+0000","last_peered":"2026-03-31T21:01:16.468907+0000","last_clean":"2026-03-31T21:01:16.468907+0000","last_became_active":"2026-03-31T21:01:05.306191+0000","last_became_peered":"2026-03-31T21:01:05.306191+0000","last_unstale":"2026-03-31T21:01:16.468907+0000","last_undegraded":"2026-03-31T21:01:16.468907+0000","last_fullsized":"2026-03-31T21:01:16.468907+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:36:03.018303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":678,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26988,"kb_used_data":156,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":159744,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10075,"internal_metadata":27449509},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":26988,"kb_used_data":156,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":159744,"data_stored":51069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":480,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":491520,"data_stored":379261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10077,"internal_metadata":27449507},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":476,"kb_used_omap":9,"kb_used_meta":26806,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":487424,"data_stored":375165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9427,"internal_metadata":27450157},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-03-31T21:01:23.845 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:recovered! 2026-03-31T21:01:23.845 DEBUG:tasks.ec_inconsistent_hinfo:create test object 2026-03-31T21:01:23.845 INFO:tasks.util.rados:rados -p unique_pool_0 put test /etc/group 2026-03-31T21:01:23.845 DEBUG:teuthology.orchestra.run.vm00:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put test /etc/group 2026-03-31T21:01:23.889 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from osd.1 shard and test deep-scrub and repair 2026-03-31T21:01:23.889 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:01:24.081 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:24.081 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":18,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:16.460993+0000","last_up_change":"2026-03-31T21:01:02.077235+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":1012853978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":1012853978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":1012853978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":1012853978}]},"public_addr":"192.168.123.100:6800/1012853978","cluster_addr":"192.168.123.100:6801/1012853978","heartbeat_back_addr":"192.168.123.100:6803/1012853978","heartbeat_front_addr":"192.168.123.100:6802/1012853978","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:01:24.095 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-03-31T21:01:24.299 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:24.312 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":18,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-03-31T21:01:24.312 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-03-31T21:01:24.313 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:01:24.340 INFO:tasks.ceph.osd.1:Stopped 2026-03-31T21:01:24.340 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 "$object" rm-attr hinfo_key 2026-03-31T21:01:24.340 DEBUG:teuthology.orchestra.run.vm00:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list |grep '"'"'"oid":"test"'"'"') 2026-03-31T21:01:24.340 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 "$object" rm-attr hinfo_key' 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -f /etc/bashrc ']' 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:+ . /etc/bashrc 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -z '' ']' 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ BASHRCSOURCED=Y 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ shopt -q login_shell 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:+++ umask 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' 0022 -eq 0 ']' 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ SHELL=/bin/bash 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorgrep.sh 2026-03-31T21:01:24.395 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'grep=grep --color=auto' 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'egrep=egrep --color=auto' 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'fgrep=fgrep --color=auto' 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorls.sh 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' '!' -t 0 ']' 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:+++ return 2026-03-31T21:01:24.397 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorsysstat.sh 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:+++ export S_COLORS=auto 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:+++ S_COLORS=auto 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorxzgrep.sh 2026-03-31T21:01:24.398 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:01:24.399 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2026-03-31T21:01:24.399 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2026-03-31T21:01:24.399 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2026-03-31T21:01:24.399 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.399 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2026-03-31T21:01:24.399 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.399 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorzgrep.sh 2026-03-31T21:01:24.400 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2026-03-31T21:01:24.400 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:01:24.401 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zgrep=zgrep --color=auto' 2026-03-31T21:01:24.401 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2026-03-31T21:01:24.401 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zegrep=zegrep --color=auto' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/debuginfod.sh 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:+++ prefix=/usr 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z /etc/keys/ima: ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset prefix 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/gawk.sh 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/lang.sh 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup 2026-03-31T21:01:24.402 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/locale 2026-03-31T21:01:24.403 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:01:24.403 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG_backup=en_US.UTF-8 2026-03-31T21:01:24.403 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:01:24.403 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /etc/locale.conf ']' 2026-03-31T21:01:24.403 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -x /usr/bin/sed ']' 2026-03-31T21:01:24.403 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ eval export LANG=en_US.UTF-8 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++++ export LANG=en_US.UTF-8 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++++ LANG=en_US.UTF-8 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG=en_US.UTF-8 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup config 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' dumb = linux ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/less.sh 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/which2.sh 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -r /proc/66360/exe ']' 2026-03-31T21:01:24.404 INFO:teuthology.orchestra.run.vm00.stderr:+++++ readlink /proc/66360/exe 2026-03-31T21:01:24.405 INFO:teuthology.orchestra.run.vm00.stderr:++++ basename /usr/bin/bash 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+++ SHELLNAME=bash 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+++ case "$SHELLNAME" in 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_declare='declare -f' 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_opt=-f 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+++ export which_declare 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+++ export -f which 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:++ unset i 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:++ unset -f pathmunge 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+ export PATH 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2026-03-31T21:01:24.406 INFO:teuthology.orchestra.run.vm00.stderr:+ unset rc 2026-03-31T21:01:24.407 INFO:teuthology.orchestra.run.vm00.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list 2026-03-31T21:01:24.407 INFO:teuthology.orchestra.run.vm00.stderr:++ grep '"oid":"test"' 2026-03-31T21:01:24.540 INFO:teuthology.orchestra.run.vm00.stderr:+ object='["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' 2026-03-31T21:01:24.540 INFO:teuthology.orchestra.run.vm00.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 '["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' rm-attr hinfo_key 2026-03-31T21:01:24.934 INFO:tasks.ceph.osd.1:Restarting daemon 2026-03-31T21:01:24.934 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-03-31T21:01:24.977 INFO:tasks.ceph.osd.1:Started 2026-03-31T21:01:24.977 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:no valid command found; 10 closest matches: 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:0 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:1 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:2 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:abort 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:assert 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:bluestore collections 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:bluestore compression stats [] 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:bluestore list [] [] 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:bluestore onode metadata 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:config diff 2026-03-31T21:01:25.053 INFO:teuthology.orchestra.run.vm00.stderr:admin_socket: invalid command 2026-03-31T21:01:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-03-31T21:01:25.056 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-03-31T21:01:25.182 INFO:tasks.ceph.osd.1.vm00.stderr:2026-03-31T21:01:25.179+0000 7fbfd8820900 -1 Falling back to public interface 2026-03-31T21:01:25.567 INFO:tasks.ceph.osd.1.vm00.stderr:2026-03-31T21:01:25.564+0000 7fbfd8820900 -1 osd.1 18 log_to_monitors true 2026-03-31T21:01:26.782 INFO:tasks.ceph.osd.1.vm00.stderr:2026-03-31T21:01:26.779+0000 7fbfcf56d640 -1 osd.1 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:01:30.056 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-03-31T21:01:30.136 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:01:30.136 INFO:teuthology.orchestra.run.vm00.stdout: "ops": [], 2026-03-31T21:01:30.136 INFO:teuthology.orchestra.run.vm00.stdout: "num_ops": 0 2026-03-31T21:01:30.136 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:01:30.144 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.1 to be up 2026-03-31T21:01:30.144 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:01:30.338 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:30.338 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":22,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:28.781960+0000","last_up_change":"2026-03-31T21:01:27.772783+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":21,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:01:30.350 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.1 is up 2026-03-31T21:01:30.350 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2026-03-31T21:01:30.558 INFO:teuthology.orchestra.run.vm00.stderr:instructing pg 2.0s0 on osd.3 to deep-scrub 2026-03-31T21:01:30.572 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 deep-scrub complete (check_time_now=1774990890) 2026-03-31T21:01:30.808 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:30.805+0000 7f516a385640 -1 log_channel(cluster) log [ERR] : 2.0 shard 1(1) soid 2:ad551702:::test:head : candidate had a read error, candidate had a missing hinfo key 2026-03-31T21:01:30.809 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:30.806+0000 7f516a385640 -1 log_channel(cluster) log [ERR] : 2.0s0 deep-scrub 0 missing, 1 inconsistent objects 2026-03-31T21:01:30.809 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:30.806+0000 7f516a385640 -1 log_channel(cluster) log [ERR] : 2.0 deep-scrub 1 errors 2026-03-31T21:01:33.822 INFO:tasks.ceph.mon.a.vm00.stderr:2026-03-31T21:01:33.819+0000 7fd709411640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 scrub errors (OSD_SCRUB_ERRORS) 2026-03-31T21:01:33.823 INFO:tasks.ceph.mon.a.vm00.stderr:2026-03-31T21:01:33.820+0000 7fd709411640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2026-03-31T21:01:35.572 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:01:35.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:01:35.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:01:35.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:01:35.695 INFO:teuthology.orchestra.run.vm00.stdout:38654705674 2026-03-31T21:01:35.696 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:35.697 INFO:teuthology.orchestra.run.vm00.stdout:38654705674 2026-03-31T21:01:35.697 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:35.702 INFO:teuthology.orchestra.run.vm00.stdout:38654705675 2026-03-31T21:01:35.702 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:35.890 INFO:teuthology.orchestra.run.vm00.stdout:90194313219 2026-03-31T21:01:35.890 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:35.915 INFO:teuthology.orchestra.run.vm00.stdout:38654705674 2026-03-31T21:01:35.929 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705674 got 38654705674 for osd.2 2026-03-31T21:01:35.930 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:35.957 INFO:teuthology.orchestra.run.vm00.stdout:38654705674 2026-03-31T21:01:35.962 INFO:teuthology.orchestra.run.vm00.stdout:38654705675 2026-03-31T21:01:35.971 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705674 got 38654705674 for osd.3 2026-03-31T21:01:35.971 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:35.983 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705675 got 38654705675 for osd.0 2026-03-31T21:01:35.983 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:36.099 INFO:teuthology.orchestra.run.vm00.stdout:90194313218 2026-03-31T21:01:36.113 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313219 got 90194313218 for osd.1 2026-03-31T21:01:37.114 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:37.317 INFO:teuthology.orchestra.run.vm00.stdout:90194313218 2026-03-31T21:01:37.330 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313219 got 90194313218 for osd.1 2026-03-31T21:01:38.331 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:38.537 INFO:teuthology.orchestra.run.vm00.stdout:90194313220 2026-03-31T21:01:38.549 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313219 got 90194313220 for osd.1 2026-03-31T21:01:38.549 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:38.549 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:38.748 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:38.749 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:38.761 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-03-31T21:01:37.793746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329548,"num_objects":4,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":39,"num_write_kb":324,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109320,"kb_used_data":1408,"kb_used_omap":46,"kb_used_meta":107793,"kb_avail":377378040,"statfs":{"total":386547056640,"available":386435112960,"internally_reserved":0,"allocated":1441792,"data_stored":927472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47190,"internal_metadata":110380970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":678,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001316"},"pg_stats":[{"pgid":"2.0","version":"18'2","reported_seq":46,"reported_epoch":22,"state":"active+clean+inconsistent","last_fresh":"2026-03-31T21:01:30.807429+0000","last_change":"2026-03-31T21:01:30.807429+0000","last_active":"2026-03-31T21:01:30.807429+0000","last_peered":"2026-03-31T21:01:30.807429+0000","last_clean":"2026-03-31T21:01:30.807429+0000","last_became_active":"2026-03-31T21:01:28.794021+0000","last_became_peered":"2026-03-31T21:01:28.794021+0000","last_unstale":"2026-03-31T21:01:30.807429+0000","last_undegraded":"2026-03-31T21:01:30.807429+0000","last_fullsized":"2026-03-31T21:01:30.807429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:30.807395+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:30.807395+0000","last_clean_scrub_stamp":"2026-03-31T21:01:12.331997+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T02:00:22.515793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":45,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-03-31T21:01:28.796183+0000","last_change":"2026-03-31T21:01:28.796087+0000","last_active":"2026-03-31T21:01:28.796183+0000","last_peered":"2026-03-31T21:01:28.796183+0000","last_clean":"2026-03-31T21:01:28.796183+0000","last_became_active":"2026-03-31T21:01:28.795870+0000","last_became_peered":"2026-03-31T21:01:28.795870+0000","last_unstale":"2026-03-31T21:01:28.796183+0000","last_undegraded":"2026-03-31T21:01:28.796183+0000","last_fullsized":"2026-03-31T21:01:28.796183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T23:20:06.743702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":1356,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2632,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27024,"kb_used_data":192,"kb_used_omap":15,"kb_used_meta":26800,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":196608,"data_stored":69820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15443,"internal_metadata":27444141},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27024,"kb_used_data":192,"kb_used_omap":12,"kb_used_meta":26803,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":196608,"data_stored":69820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12694,"internal_metadata":27446890},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":516,"kb_used_omap":6,"kb_used_meta":27385,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":528384,"data_stored":398012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6359,"internal_metadata":28043049},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":508,"kb_used_omap":12,"kb_used_meta":26803,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":520192,"data_stored":389820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12694,"internal_metadata":27446890},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2632,"internal_metadata":0}]}} 2026-03-31T21:01:38.761 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "18'2", 'reported_seq': 46, 'reported_epoch': 22, 'state': 'active+clean+inconsistent', 'last_fresh': '2026-03-31T21:01:30.807429+0000', 'last_change': '2026-03-31T21:01:30.807429+0000', 'last_active': '2026-03-31T21:01:30.807429+0000', 'last_peered': '2026-03-31T21:01:30.807429+0000', 'last_clean': '2026-03-31T21:01:30.807429+0000', 'last_became_active': '2026-03-31T21:01:28.794021+0000', 'last_became_peered': '2026-03-31T21:01:28.794021+0000', 'last_unstale': '2026-03-31T21:01:30.807429+0000', 'last_undegraded': '2026-03-31T21:01:30.807429+0000', 'last_fullsized': '2026-03-31T21:01:30.807429+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 22, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:30.807395+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:30.807395+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:12.331997+0000', 'objects_scrubbed': 2, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-04-02T02:00:22.515793+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1356, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 1, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 1, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:01:38.761 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg repair 2.0 2026-03-31T21:01:38.958 INFO:teuthology.orchestra.run.vm00.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-03-31T21:01:38.970 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 deep-scrub complete (check_time_now=1774990898) 2026-03-31T21:01:39.871 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:39.868+0000 7f5169383640 -1 log_channel(cluster) log [ERR] : 2.0 shard 1(1) soid 2:ad551702:::test:head : candidate had a read error, candidate had a missing hinfo key 2026-03-31T21:01:39.872 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:39.868+0000 7f5169383640 -1 log_channel(cluster) log [ERR] : 2.0s0 repair 0 missing, 1 inconsistent objects 2026-03-31T21:01:39.872 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:39.868+0000 7f5169383640 -1 log_channel(cluster) log [ERR] : 2.0s0 repair 0 missing, 1 inconsistent objects 2026-03-31T21:01:39.872 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:01:39.868+0000 7f5169383640 -1 log_channel(cluster) log [ERR] : 2.0 repair 1 errors, 1 fixed 2026-03-31T21:01:43.971 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:01:43.971 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:01:43.971 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:01:43.971 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:01:44.085 INFO:teuthology.orchestra.run.vm00.stdout:38654705678 2026-03-31T21:01:44.085 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:44.093 INFO:teuthology.orchestra.run.vm00.stdout:38654705677 2026-03-31T21:01:44.093 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:44.094 INFO:teuthology.orchestra.run.vm00.stdout:38654705677 2026-03-31T21:01:44.094 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:44.098 INFO:teuthology.orchestra.run.vm00.stdout:90194313222 2026-03-31T21:01:44.098 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:44.311 INFO:teuthology.orchestra.run.vm00.stdout:38654705677 2026-03-31T21:01:44.322 INFO:teuthology.orchestra.run.vm00.stdout:38654705676 2026-03-31T21:01:44.329 INFO:teuthology.orchestra.run.vm00.stdout:38654705676 2026-03-31T21:01:44.332 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705677 for osd.0 2026-03-31T21:01:44.340 INFO:teuthology.orchestra.run.vm00.stdout:90194313221 2026-03-31T21:01:44.342 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705677 got 38654705676 for osd.3 2026-03-31T21:01:44.347 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705677 got 38654705676 for osd.2 2026-03-31T21:01:44.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313222 got 90194313221 for osd.1 2026-03-31T21:01:45.333 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:45.343 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:45.348 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:45.356 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:45.569 INFO:teuthology.orchestra.run.vm00.stdout:38654705677 2026-03-31T21:01:45.585 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705677 for osd.0 2026-03-31T21:01:45.603 INFO:teuthology.orchestra.run.vm00.stdout:90194313221 2026-03-31T21:01:45.609 INFO:teuthology.orchestra.run.vm00.stdout:38654705676 2026-03-31T21:01:45.616 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313222 got 90194313221 for osd.1 2026-03-31T21:01:45.625 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705677 got 38654705676 for osd.2 2026-03-31T21:01:45.655 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:45.652+0000 7f69a9d74640 0 --2- 192.168.123.100:0/262519906 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f6988024380 0x7f6988044800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:01:45.774 INFO:teuthology.orchestra.run.vm00.stdout:38654705676 2026-03-31T21:01:45.786 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705677 got 38654705676 for osd.3 2026-03-31T21:01:46.586 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:01:46.616 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:01:46.626 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:01:46.787 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:01:46.811 INFO:teuthology.orchestra.run.vm00.stdout:38654705678 2026-03-31T21:01:46.826 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705678 for osd.0 2026-03-31T21:01:46.826 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:46.845 INFO:teuthology.orchestra.run.vm00.stdout:38654705677 2026-03-31T21:01:46.859 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705677 got 38654705677 for osd.2 2026-03-31T21:01:46.859 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:46.870 INFO:teuthology.orchestra.run.vm00.stdout:90194313222 2026-03-31T21:01:46.884 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313222 got 90194313222 for osd.1 2026-03-31T21:01:46.885 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:47.015 INFO:teuthology.orchestra.run.vm00.stdout:38654705677 2026-03-31T21:01:47.028 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705677 got 38654705677 for osd.3 2026-03-31T21:01:47.029 DEBUG:teuthology.parallel:result is None 2026-03-31T21:01:47.029 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:01:47.229 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:01:47.230 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:01:47.242 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2026-03-31T21:01:45.794796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329548,"num_objects":4,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":39,"num_write_kb":324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":109308,"kb_used_data":1404,"kb_used_omap":46,"kb_used_meta":107793,"kb_avail":377378052,"statfs":{"total":386547056640,"available":386435125248,"internally_reserved":0,"allocated":1437696,"data_stored":923376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47397,"internal_metadata":110380763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":-1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":-1,"num_objects_recovered":1,"num_bytes_recovered":678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001627"},"pg_stats":[{"pgid":"2.0","version":"18'2","reported_seq":61,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-03-31T21:01:40.073077+0000","last_change":"2026-03-31T21:01:40.073663+0000","last_active":"2026-03-31T21:01:40.073077+0000","last_peered":"2026-03-31T21:01:40.073077+0000","last_clean":"2026-03-31T21:01:40.073077+0000","last_became_active":"2026-03-31T21:01:28.794021+0000","last_became_peered":"2026-03-31T21:01:28.794021+0000","last_unstale":"2026-03-31T21:01:40.073077+0000","last_undegraded":"2026-03-31T21:01:40.073077+0000","last_fullsized":"2026-03-31T21:01:40.073077+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T07:23:04.740520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1356,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":45,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-03-31T21:01:28.796183+0000","last_change":"2026-03-31T21:01:28.796087+0000","last_active":"2026-03-31T21:01:28.796183+0000","last_peered":"2026-03-31T21:01:28.796183+0000","last_clean":"2026-03-31T21:01:28.796183+0000","last_became_active":"2026-03-31T21:01:28.795870+0000","last_became_peered":"2026-03-31T21:01:28.795870+0000","last_unstale":"2026-03-31T21:01:28.796183+0000","last_undegraded":"2026-03-31T21:01:28.796183+0000","last_fullsized":"2026-03-31T21:01:28.796183+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T23:20:06.743702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":1356,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2244,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705677,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27024,"kb_used_data":192,"kb_used_omap":15,"kb_used_meta":26800,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":196608,"data_stored":69820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15650,"internal_metadata":27443934},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705677,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27024,"kb_used_data":192,"kb_used_omap":12,"kb_used_meta":26803,"kb_avail":94344816,"statfs":{"total":96636764160,"available":96609091584,"internally_reserved":0,"allocated":196608,"data_stored":69820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12694,"internal_metadata":27446890},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":512,"kb_used_omap":6,"kb_used_meta":27385,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":524288,"data_stored":393916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6359,"internal_metadata":28043049},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705678,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":508,"kb_used_omap":12,"kb_used_meta":26803,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":520192,"data_stored":389820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12694,"internal_metadata":27446890},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2244,"internal_metadata":0}]}} 2026-03-31T21:01:47.243 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "18'2", 'reported_seq': 61, 'reported_epoch': 22, 'state': 'active+clean', 'last_fresh': '2026-03-31T21:01:40.073077+0000', 'last_change': '2026-03-31T21:01:40.073663+0000', 'last_active': '2026-03-31T21:01:40.073077+0000', 'last_peered': '2026-03-31T21:01:40.073077+0000', 'last_clean': '2026-03-31T21:01:40.073077+0000', 'last_became_active': '2026-03-31T21:01:28.794021+0000', 'last_became_peered': '2026-03-31T21:01:28.794021+0000', 'last_unstale': '2026-03-31T21:01:40.073077+0000', 'last_undegraded': '2026-03-31T21:01:40.073077+0000', 'last_fullsized': '2026-03-31T21:01:40.073077+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 22, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-04-02T07:23:04.740520+0000', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1356, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 678, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:01:47.243 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from primary osd.3 shard and test backfill 2026-03-31T21:01:47.243 DEBUG:tasks.ec_inconsistent_hinfo:write some data 2026-03-31T21:01:47.243 INFO:tasks.util.rados:rados -p unique_pool_0 bench 30 write -b 4096 --no-cleanup 2026-03-31T21:01:47.243 DEBUG:teuthology.orchestra.run.vm00:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 30 write -b 4096 --no-cleanup 2026-03-31T21:01:47.284 INFO:teuthology.orchestra.run.vm00.stdout:hints = 1 2026-03-31T21:01:47.284 INFO:teuthology.orchestra.run.vm00.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 8192 for up to 30 seconds or 0 objects 2026-03-31T21:01:47.285 INFO:teuthology.orchestra.run.vm00.stdout:Object prefix: benchmark_data_vm00.local_69782 2026-03-31T21:01:47.285 INFO:teuthology.orchestra.run.vm00.stderr:INFO: op_size has been rounded to 8192 2026-03-31T21:01:47.285 INFO:teuthology.orchestra.run.vm00.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-03-31T21:01:47.285 INFO:teuthology.orchestra.run.vm00.stdout: 0 0 0 0 0 0 - 0 2026-03-31T21:01:47.374 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:47.371+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x562736acd100 msgr2=0x562736aed5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:47.386 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:47.383+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074060570 msgr2=0x7f8074080990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:47.491 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:47.488+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074060570 msgr2=0x7f8074080990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:47.491 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:47.488+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074060570 msgr2=0x7f8074080990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:47.492 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:47.489+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x562736acd100 msgr2=0x7f80740833a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:47.538 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:47.535+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074060570 msgr2=0x7f8074083830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:48.085 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.082+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074060570 msgr2=0x7f80740837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:48.206 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.203+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082660 msgr2=0x7f80740837d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:48.218 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.215+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074081e90 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:48.284 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.281+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074080950 msgr2=0x7f8074081350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:48.285 INFO:teuthology.orchestra.run.vm00.stdout: 1 15 1292 1277 9.97579 9.97656 0.0160134 0.0123876 2026-03-31T21:01:48.501 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.497+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740837d0 msgr2=0x7f8074082e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:48.501 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.498+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074080950 msgr2=0x7f8074081350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:48.727 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.724+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074080950 msgr2=0x7f8074081350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:48.843 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:48.840+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740837d0 msgr2=0x7f8074082e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:49.242 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:49.239+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740837d0 msgr2=0x7f8074082e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:49.285 INFO:teuthology.orchestra.run.vm00.stdout: 2 16 2165 2149 8.39391 6.8125 0.011248 0.0148487 2026-03-31T21:01:49.320 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:49.317+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083360 msgr2=0x7f8074080f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:49.522 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:49.519+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083360 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:49.553 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:49.550+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740837d0 msgr2=0x7f8074082e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:49.586 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:49.583+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083360 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:49.643 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:49.640+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083360 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:50.012 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:50.009+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f8074083730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:50.110 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:50.107+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f8074083730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:50.285 INFO:teuthology.orchestra.run.vm00.stdout: 3 16 3117 3101 8.0749 7.4375 0.0148711 0.0154273 2026-03-31T21:01:50.374 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:50.371+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f8074083730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:50.381 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:50.378+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a59d0 msgr2=0x7f8074081380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:50.391 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:50.388+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f8074082e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:50.398 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:50.396+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a55b0 msgr2=0x7f8074083410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:50.480 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:50.477+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083410 msgr2=0x7f8074082e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.274 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.271+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083410 msgr2=0x7f80740838e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:51.285 INFO:teuthology.orchestra.run.vm00.stdout: 4 16 3568 3552 6.93686 3.52344 0.0124688 0.0179909 2026-03-31T21:01:51.296 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.293+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a46a0 msgr2=0x7f8074082e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.298 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.295+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083410 msgr2=0x7f8074083880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:51.333 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.330+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074080950 msgr2=0x7f80740a4e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.422 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.419+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f8074083950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.442 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.439+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f8074083410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.443 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.440+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f8074083950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.580 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.577+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f8074083410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.619 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.616+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f8074083410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:51.955 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:51.952+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f8074083870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:52.059 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:52.056+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f8074083330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:52.071 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:52.068+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f8074083870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:52.285 INFO:teuthology.orchestra.run.vm00.stdout: 5 16 4603 4587 7.16652 8.08594 0.00854717 0.0168755 2026-03-31T21:01:52.538 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:52.535+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740831f0 msgr2=0x7f80740a42e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:52.578 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:52.575+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a7620 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:52.718 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:52.715+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083d10 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:01:52.855 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:52.852+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a53c0 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:52.869 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:52.866+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074083d10 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:53.285 INFO:teuthology.orchestra.run.vm00.stdout: 6 16 5775 5759 7.4979 9.15625 0.0152818 0.0166192 2026-03-31T21:01:54.285 INFO:teuthology.orchestra.run.vm00.stdout: 7 16 6244 6228 6.95018 3.66406 0.00588259 0.0179644 2026-03-31T21:01:54.316 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:54.313+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a42e0 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:55.285 INFO:teuthology.orchestra.run.vm00.stdout: 8 16 6405 6389 6.23863 1.25781 0.089202 0.0199807 2026-03-31T21:01:55.882 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:55.879+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:55.911 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:55.908+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a42e0 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:55.951 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:01:55.948+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:01:56.285 INFO:teuthology.orchestra.run.vm00.stdout: 9 16 6956 6940 6.0237 4.30469 0.0038538 0.0199827 2026-03-31T21:01:57.285 INFO:teuthology.orchestra.run.vm00.stdout: 10 16 6956 6940 5.42133 0 - 0.0199827 2026-03-31T21:01:58.285 INFO:teuthology.orchestra.run.vm00.stdout: 11 16 6956 6940 4.92848 0 - 0.0199827 2026-03-31T21:01:59.286 INFO:teuthology.orchestra.run.vm00.stdout: 12 16 6956 6940 4.51777 0 - 0.0199827 2026-03-31T21:02:00.286 INFO:teuthology.orchestra.run.vm00.stdout: 13 16 6956 6940 4.17025 0 - 0.0199827 2026-03-31T21:02:01.286 INFO:teuthology.orchestra.run.vm00.stdout: 14 16 6956 6940 3.87238 0 - 0.0199827 2026-03-31T21:02:02.286 INFO:teuthology.orchestra.run.vm00.stdout: 15 16 6956 6940 3.61422 0 - 0.0199827 2026-03-31T21:02:03.286 INFO:teuthology.orchestra.run.vm00.stdout: 16 16 6956 6940 3.38833 0 - 0.0199827 2026-03-31T21:02:04.286 INFO:teuthology.orchestra.run.vm00.stdout: 17 16 6956 6940 3.18902 0 - 0.0199827 2026-03-31T21:02:05.286 INFO:teuthology.orchestra.run.vm00.stdout: 18 16 6956 6940 3.01185 0 - 0.0199827 2026-03-31T21:02:06.286 INFO:teuthology.orchestra.run.vm00.stdout: 19 16 6956 6940 2.85333 0 - 0.0199827 2026-03-31T21:02:07.287 INFO:teuthology.orchestra.run.vm00.stdout:2026-03-31T21:02:07.285140+0000 min lat: 0.00272537 max lat: 0.617807 avg lat: 0.0199827 2026-03-31T21:02:07.287 INFO:teuthology.orchestra.run.vm00.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-03-31T21:02:07.287 INFO:teuthology.orchestra.run.vm00.stdout: 20 16 6956 6940 2.71066 0 - 0.0199827 2026-03-31T21:02:08.287 INFO:teuthology.orchestra.run.vm00.stdout: 21 16 6956 6940 2.58158 0 - 0.0199827 2026-03-31T21:02:09.287 INFO:teuthology.orchestra.run.vm00.stdout: 22 16 6956 6940 2.46424 0 - 0.0199827 2026-03-31T21:02:10.287 INFO:teuthology.orchestra.run.vm00.stdout: 23 16 6956 6940 2.3571 0 - 0.0199827 2026-03-31T21:02:11.056 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.053+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:11.193 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.190+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:11.212 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.209+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a42e0 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:11.287 INFO:teuthology.orchestra.run.vm00.stdout: 24 16 7423 7407 2.41088 0.243229 0.00715679 0.0518337 2026-03-31T21:02:11.333 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.329+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:11.336 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.333+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a42e0 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:11.350 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.347+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:11.748 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.745+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:11.782 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:11.779+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a42e0 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:12.111 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:12.108+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a42e0 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:12.219 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:12.216+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:12.287 INFO:teuthology.orchestra.run.vm00.stdout: 25 16 8746 8730 2.72785 10.3359 0.0107852 0.0458101 2026-03-31T21:02:12.349 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:12.346+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a42e0 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:12.495 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:12.492+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:12.564 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:12.561+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:12.589 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:12.586+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:12.872 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:12.869+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:13.122 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:13.119+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:13.124 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:13.121+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:13.184 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:13.181+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:13.287 INFO:teuthology.orchestra.run.vm00.stdout: 26 16 9709 9693 2.91226 7.52344 0.0138179 0.0427661 2026-03-31T21:02:13.534 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:13.531+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:13.747 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:13.744+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:14.056 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:14.053+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:14.089 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:14.086+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:14.133 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:14.130+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:14.287 INFO:teuthology.orchestra.run.vm00.stdout: 27 16 10856 10840 3.13625 8.96094 0.0162407 0.0398411 2026-03-31T21:02:14.413 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:14.410+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:14.434 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:14.431+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:14.822 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:14.819+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:15.287 INFO:teuthology.orchestra.run.vm00.stdout: 28 16 11677 11661 3.2533 6.41406 0.0139841 0.0384097 2026-03-31T21:02:15.700 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:15.697+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f80740835c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:15.707 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:15.704+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6a40 msgr2=0x7f8074080950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:15.896 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:15.893+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:15.896 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:15.894+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740a6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:16.152 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:16.149+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740a6f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:16.204 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:16.200+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:16.250 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:16.247+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:16.263 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:16.259+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2026-03-31T21:02:16.287 INFO:teuthology.orchestra.run.vm00.stdout: 29 16 12654 12638 3.40429 7.63281 0.0104564 0.0367027 2026-03-31T21:02:16.292 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:16.289+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f80740a6290 msgr2=0x7f80740a6f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:16.802 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:02:16.799+0000 7f8096eca640 0 -- 192.168.123.100:0/1007484636 >> v2:192.168.123.100:6804/4053481465 conn(0x7f8074082e20 msgr2=0x7f80740a6a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2026-03-31T21:02:17.287 INFO:teuthology.orchestra.run.vm00.stdout: 30 12 13364 13352 3.47673 5.57812 0.0135436 0.0359434 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Total time run: 30.0096 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Total writes made: 13364 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Write size: 8192 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Object size: 8192 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Bandwidth (MB/sec): 3.4791 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Stddev Bandwidth: 3.8566 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Max bandwidth (MB/sec): 10.3359 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Min bandwidth (MB/sec): 0 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Average IOPS: 445 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Stddev IOPS: 493.648 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Max IOPS: 1323 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Min IOPS: 0 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Average Latency(s): 0.0359232 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Stddev Latency(s): 0.520094 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Max latency(s): 15.0068 2026-03-31T21:02:18.288 INFO:teuthology.orchestra.run.vm00.stdout:Min latency(s): 0.00272537 2026-03-31T21:02:18.293 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:02:18.531 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:02:18.531 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":22,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:01:28.781960+0000","last_up_change":"2026-03-31T21:01:27.772783+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":21,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":4053481465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":4053481465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":4053481465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":4053481465}]},"public_addr":"192.168.123.100:6804/4053481465","cluster_addr":"192.168.123.100:6805/4053481465","heartbeat_back_addr":"192.168.123.100:6807/4053481465","heartbeat_front_addr":"192.168.123.100:6806/4053481465","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:02:18.542 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-03-31T21:02:18.733 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:02:18.744 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":22,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-03-31T21:02:18.744 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-03-31T21:02:18.744 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:02:18.795 INFO:tasks.ceph.osd.3:Stopped 2026-03-31T21:02:18.796 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key 2026-03-31T21:02:18.796 DEBUG:teuthology.orchestra.run.vm00:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"'"'"oid":"test"'"'"') 2026-03-31T21:02:18.796 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key' 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -f /etc/bashrc ']' 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:+ . /etc/bashrc 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -z '' ']' 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ BASHRCSOURCED=Y 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ shopt -q login_shell 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:+++ umask 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' 0022 -eq 0 ']' 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ SHELL=/bin/bash 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorgrep.sh 2026-03-31T21:02:18.810 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'grep=grep --color=auto' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'egrep=egrep --color=auto' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'fgrep=fgrep --color=auto' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorls.sh 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' '!' -t 0 ']' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ return 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorsysstat.sh 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ export S_COLORS=auto 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ S_COLORS=auto 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorxzgrep.sh 2026-03-31T21:02:18.812 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorzgrep.sh 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2026-03-31T21:02:18.814 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zgrep=zgrep --color=auto' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zegrep=zegrep --color=auto' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/debuginfod.sh 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ prefix=/usr 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z /etc/keys/ima: ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset prefix 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/gawk.sh 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/lang.sh 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/locale 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG_backup=en_US.UTF-8 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /etc/locale.conf ']' 2026-03-31T21:02:18.816 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -x /usr/bin/sed ']' 2026-03-31T21:02:18.817 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ eval export LANG=en_US.UTF-8 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++++ export LANG=en_US.UTF-8 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++++ LANG=en_US.UTF-8 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG=en_US.UTF-8 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup config 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' dumb = linux ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/less.sh 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/which2.sh 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -r /proc/69925/exe ']' 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:+++++ readlink /proc/69925/exe 2026-03-31T21:02:18.818 INFO:teuthology.orchestra.run.vm00.stderr:++++ basename /usr/bin/bash 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+++ SHELLNAME=bash 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+++ case "$SHELLNAME" in 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_declare='declare -f' 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_opt=-f 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+++ export which_declare 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+++ export -f which 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:++ unset i 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:++ unset -f pathmunge 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+ export PATH 2026-03-31T21:02:18.819 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2026-03-31T21:02:18.820 INFO:teuthology.orchestra.run.vm00.stderr:+ unset rc 2026-03-31T21:02:18.820 INFO:teuthology.orchestra.run.vm00.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list 2026-03-31T21:02:18.820 INFO:teuthology.orchestra.run.vm00.stderr:++ grep '"oid":"test"' 2026-03-31T21:02:19.079 INFO:teuthology.orchestra.run.vm00.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-03-31T21:02:19.080 INFO:teuthology.orchestra.run.vm00.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-03-31T21:02:19.302 INFO:teuthology.orchestra.run.vm00.stderr:+ object='["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' 2026-03-31T21:02:19.302 INFO:teuthology.orchestra.run.vm00.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 '["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' rm-attr hinfo_key 2026-03-31T21:02:20.449 INFO:tasks.ceph.osd.3:Restarting daemon 2026-03-31T21:02:20.449 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-03-31T21:02:20.491 INFO:tasks.ceph.osd.3:Started 2026-03-31T21:02:20.491 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-03-31T21:02:20.562 INFO:teuthology.orchestra.run.vm00.stderr:no valid command found; 10 closest matches: 2026-03-31T21:02:20.562 INFO:teuthology.orchestra.run.vm00.stderr:0 2026-03-31T21:02:20.562 INFO:teuthology.orchestra.run.vm00.stderr:1 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:2 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:abort 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:assert 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:bluestore collections 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:bluestore compression stats [] 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:bluestore list [] [] 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:bluestore onode metadata 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:config diff 2026-03-31T21:02:20.563 INFO:teuthology.orchestra.run.vm00.stderr:admin_socket: invalid command 2026-03-31T21:02:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-03-31T21:02:20.565 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-03-31T21:02:20.720 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:02:20.717+0000 7f4f334ba900 -1 Falling back to public interface 2026-03-31T21:02:21.136 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:02:21.133+0000 7f4f334ba900 -1 osd.3 22 log_to_monitors true 2026-03-31T21:02:22.190 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:02:22.187+0000 7f4f2a209640 -1 osd.3 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:02:25.566 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-03-31T21:02:25.638 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:02:25.638 INFO:teuthology.orchestra.run.vm00.stdout: "ops": [], 2026-03-31T21:02:25.638 INFO:teuthology.orchestra.run.vm00.stdout: "num_ops": 0 2026-03-31T21:02:25.638 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:02:25.644 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.3 to be up 2026-03-31T21:02:25.644 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:02:25.834 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:02:25.834 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":27,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:02:24.189373+0000","last_up_change":"2026-03-31T21:02:23.181592+0000","last_in_change":"2026-03-31T21:00:58.239148+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":23,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":26,"up_thru":26,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":2551005687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":2551005687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":2551005687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":2551005687}]},"public_addr":"192.168.123.100:6804/2551005687","cluster_addr":"192.168.123.100:6805/2551005687","heartbeat_back_addr":"192.168.123.100:6807/2551005687","heartbeat_front_addr":"192.168.123.100:6806/2551005687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":23}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:02:25.846 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.3 is up 2026-03-31T21:02:25.846 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2026-03-31T21:02:26.042 INFO:tasks.ceph.mon.a.vm00.stderr:2026-03-31T21:02:26.039+0000 7fd706c0c640 -1 mon.a@0(leader).osd e27 definitely_dead 0 2026-03-31T21:02:26.210 INFO:teuthology.orchestra.run.vm00.stderr:marked out osd.1. 2026-03-31T21:02:26.222 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.1 to osd.0 complete 2026-03-31T21:02:31.223 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:02:31.223 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:02:31.223 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:02:31.223 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:02:31.370 INFO:teuthology.orchestra.run.vm00.stdout:111669149699 2026-03-31T21:02:31.371 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:31.391 INFO:teuthology.orchestra.run.vm00.stdout:90194313232 2026-03-31T21:02:31.391 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:31.394 INFO:teuthology.orchestra.run.vm00.stdout:38654705688 2026-03-31T21:02:31.394 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:31.395 INFO:teuthology.orchestra.run.vm00.stdout:38654705687 2026-03-31T21:02:31.396 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:31.610 INFO:teuthology.orchestra.run.vm00.stdout:38654705687 2026-03-31T21:02:31.610 INFO:teuthology.orchestra.run.vm00.stdout:90194313231 2026-03-31T21:02:31.614 INFO:teuthology.orchestra.run.vm00.stdout:111669149698 2026-03-31T21:02:31.622 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313232 got 90194313231 for osd.1 2026-03-31T21:02:31.622 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705688 got 38654705687 for osd.0 2026-03-31T21:02:31.629 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149699 got 111669149698 for osd.3 2026-03-31T21:02:31.655 INFO:teuthology.orchestra.run.vm00.stdout:38654705686 2026-03-31T21:02:31.669 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705687 got 38654705686 for osd.2 2026-03-31T21:02:32.623 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:32.623 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:32.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:32.669 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:32.951 INFO:teuthology.orchestra.run.vm00.stdout:38654705687 2026-03-31T21:02:32.957 INFO:teuthology.orchestra.run.vm00.stdout:38654705688 2026-03-31T21:02:32.965 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705687 got 38654705687 for osd.2 2026-03-31T21:02:32.965 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:32.984 INFO:teuthology.orchestra.run.vm00.stdout:90194313233 2026-03-31T21:02:32.985 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705688 got 38654705688 for osd.0 2026-03-31T21:02:32.985 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:32.998 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313232 got 90194313233 for osd.1 2026-03-31T21:02:32.998 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:33.005 INFO:teuthology.orchestra.run.vm00.stdout:111669149699 2026-03-31T21:02:33.018 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149699 got 111669149699 for osd.3 2026-03-31T21:02:33.018 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:33.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:02:33.322 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:02:33.323 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:02:33.335 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2026-03-31T21:02:31.800625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11713,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1658,"num_bytes_recovered":13886630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":214772,"kb_used_data":113284,"kb_used_omap":43,"kb_used_meta":101396,"kb_avail":282900748,"statfs":{"total":289910292480,"available":289690365952,"internally_reserved":0,"allocated":116002816,"data_stored":115503702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44185,"internal_metadata":103830375},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-13367,"num_objects_misplaced":11713,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1657,"num_bytes_recovered":13885952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000679"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":16762,"reported_epoch":30,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:02:31.352952+0000","last_change":"2026-03-31T21:02:27.877871+0000","last_active":"2026-03-31T21:02:31.352952+0000","last_peered":"2026-03-31T21:02:31.352952+0000","last_clean":"2026-03-31T21:02:24.201524+0000","last_became_active":"2026-03-31T21:02:27.876634+0000","last_became_peered":"2026-03-31T21:02:27.876634+0000","last_unstale":"2026-03-31T21:02:31.352952+0000","last_undegraded":"2026-03-31T21:02:31.352952+0000","last_fullsized":"2026-03-31T21:02:31.352952+0000","mapping_epoch":29,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11713,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1656,"num_bytes_recovered":13558438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-03-31T21:02:27.863888+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:02:27.863888+0000","last_peered":"2026-03-31T21:02:27.863888+0000","last_clean":"2026-03-31T21:02:27.863888+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:02:27.863888+0000","last_undegraded":"2026-03-31T21:02:27.863888+0000","last_fullsized":"2026-03-31T21:02:27.863888+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11713,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1656,"num_bytes_recovered":13558438,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171048960,"data_stored":171048960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149699,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83492,"kb_used_data":53716,"kb_used_omap":8,"kb_used_meta":29751,"kb_avail":94288348,"statfs":{"total":96636764160,"available":96551268352,"internally_reserved":0,"allocated":55005184,"data_stored":54841202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8381,"internal_metadata":30465859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98920,"kb_used_data":54040,"kb_used_omap":17,"kb_used_meta":44846,"kb_avail":94272920,"statfs":{"total":96636764160,"available":96535470080,"internally_reserved":0,"allocated":55336960,"data_stored":55169394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17902,"internal_metadata":45922834},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313233,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":5528,"kb_used_omap":17,"kb_used_meta":26798,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":5660672,"data_stored":5493106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17902,"internal_metadata":27441682},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6803456,"data_stored":6803456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:02:33.335 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 16762, 'reported_epoch': 30, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:02:31.352952+0000', 'last_change': '2026-03-31T21:02:27.877871+0000', 'last_active': '2026-03-31T21:02:31.352952+0000', 'last_peered': '2026-03-31T21:02:31.352952+0000', 'last_clean': '2026-03-31T21:02:24.201524+0000', 'last_became_active': '2026-03-31T21:02:27.876634+0000', 'last_became_peered': '2026-03-31T21:02:27.876634+0000', 'last_unstale': '2026-03-31T21:02:31.352952+0000', 'last_undegraded': '2026-03-31T21:02:31.352952+0000', 'last_fullsized': '2026-03-31T21:02:31.352952+0000', 'mapping_epoch': 29, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 27, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11713, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1656, 'num_bytes_recovered': 13558438, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:02:33.335 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:02:38.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:02:38.337 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:02:38.337 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:02:38.337 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:02:38.495 INFO:teuthology.orchestra.run.vm00.stdout:90194313235 2026-03-31T21:02:38.495 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:38.509 INFO:teuthology.orchestra.run.vm00.stdout:111669149702 2026-03-31T21:02:38.510 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:38.510 INFO:teuthology.orchestra.run.vm00.stdout:38654705691 2026-03-31T21:02:38.510 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:38.513 INFO:teuthology.orchestra.run.vm00.stdout:38654705691 2026-03-31T21:02:38.514 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:38.752 INFO:teuthology.orchestra.run.vm00.stdout:111669149701 2026-03-31T21:02:38.754 INFO:teuthology.orchestra.run.vm00.stdout:38654705689 2026-03-31T21:02:38.755 INFO:teuthology.orchestra.run.vm00.stdout:38654705689 2026-03-31T21:02:38.762 INFO:teuthology.orchestra.run.vm00.stdout:90194313234 2026-03-31T21:02:38.767 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705691 got 38654705689 for osd.0 2026-03-31T21:02:38.770 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149702 got 111669149701 for osd.3 2026-03-31T21:02:38.770 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705691 got 38654705689 for osd.2 2026-03-31T21:02:38.775 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313235 got 90194313234 for osd.1 2026-03-31T21:02:39.768 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:39.770 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:39.770 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:39.776 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:40.075 INFO:teuthology.orchestra.run.vm00.stdout:38654705691 2026-03-31T21:02:40.078 INFO:teuthology.orchestra.run.vm00.stdout:111669149702 2026-03-31T21:02:40.094 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705691 got 38654705691 for osd.0 2026-03-31T21:02:40.094 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:40.096 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149702 got 111669149702 for osd.3 2026-03-31T21:02:40.096 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:40.162 INFO:teuthology.orchestra.run.vm00.stdout:90194313235 2026-03-31T21:02:40.162 INFO:teuthology.orchestra.run.vm00.stdout:38654705691 2026-03-31T21:02:40.175 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705691 got 38654705691 for osd.2 2026-03-31T21:02:40.176 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:40.176 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313235 got 90194313235 for osd.1 2026-03-31T21:02:40.176 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:40.176 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:02:40.439 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:02:40.439 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:02:40.455 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2026-03-31T21:02:39.801648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9015,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4356,"num_bytes_recovered":35981132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":226072,"kb_used_data":124584,"kb_used_omap":43,"kb_used_meta":101396,"kb_avail":282889448,"statfs":{"total":289910292480,"available":289678794752,"internally_reserved":0,"allocated":127574016,"data_stored":127074902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44185,"internal_metadata":103830375},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9015,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4355,"num_bytes_recovered":35980454,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.940486"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":22158,"reported_epoch":30,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:02:38.486513+0000","last_change":"2026-03-31T21:02:27.877871+0000","last_active":"2026-03-31T21:02:38.486513+0000","last_peered":"2026-03-31T21:02:38.486513+0000","last_clean":"2026-03-31T21:02:24.201524+0000","last_became_active":"2026-03-31T21:02:27.876634+0000","last_became_peered":"2026-03-31T21:02:27.876634+0000","last_unstale":"2026-03-31T21:02:38.486513+0000","last_undegraded":"2026-03-31T21:02:38.486513+0000","last_fullsized":"2026-03-31T21:02:38.486513+0000","mapping_epoch":29,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9015,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4354,"num_bytes_recovered":35652940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":30,"state":"active+clean","last_fresh":"2026-03-31T21:02:27.863888+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:02:27.863888+0000","last_peered":"2026-03-31T21:02:27.863888+0000","last_clean":"2026-03-31T21:02:27.863888+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:02:27.863888+0000","last_undegraded":"2026-03-31T21:02:27.863888+0000","last_fullsized":"2026-03-31T21:02:27.863888+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9015,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4354,"num_bytes_recovered":35652940,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":182091776,"data_stored":182091776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83492,"kb_used_data":53716,"kb_used_omap":8,"kb_used_meta":29751,"kb_avail":94288348,"statfs":{"total":96636764160,"available":96551268352,"internally_reserved":0,"allocated":55005184,"data_stored":54841202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8381,"internal_metadata":30465859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98920,"kb_used_data":54040,"kb_used_omap":17,"kb_used_meta":44846,"kb_avail":94272920,"statfs":{"total":96636764160,"available":96535470080,"internally_reserved":0,"allocated":55336960,"data_stored":55169394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17902,"internal_metadata":45922834},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313235,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705691,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43660,"kb_used_data":16828,"kb_used_omap":17,"kb_used_meta":26798,"kb_avail":94328180,"statfs":{"total":96636764160,"available":96592056320,"internally_reserved":0,"allocated":17231872,"data_stored":17064306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17902,"internal_metadata":27441682},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17846272,"data_stored":17846272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:02:40.456 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 22158, 'reported_epoch': 30, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:02:38.486513+0000', 'last_change': '2026-03-31T21:02:27.877871+0000', 'last_active': '2026-03-31T21:02:38.486513+0000', 'last_peered': '2026-03-31T21:02:38.486513+0000', 'last_clean': '2026-03-31T21:02:24.201524+0000', 'last_became_active': '2026-03-31T21:02:27.876634+0000', 'last_became_peered': '2026-03-31T21:02:27.876634+0000', 'last_unstale': '2026-03-31T21:02:38.486513+0000', 'last_undegraded': '2026-03-31T21:02:38.486513+0000', 'last_fullsized': '2026-03-31T21:02:38.486513+0000', 'mapping_epoch': 29, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 27, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9015, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4354, 'num_bytes_recovered': 35652940, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:02:40.456 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:02:45.456 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:02:45.457 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:02:45.457 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:02:45.457 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:02:45.612 INFO:teuthology.orchestra.run.vm00.stdout:38654705693 2026-03-31T21:02:45.612 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:45.630 INFO:teuthology.orchestra.run.vm00.stdout:111669149704 2026-03-31T21:02:45.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:45.640 INFO:teuthology.orchestra.run.vm00.stdout:90194313237 2026-03-31T21:02:45.641 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:45.671 INFO:teuthology.orchestra.run.vm00.stdout:38654705693 2026-03-31T21:02:45.671 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:45.961 INFO:teuthology.orchestra.run.vm00.stdout:90194313236 2026-03-31T21:02:45.977 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313237 got 90194313236 for osd.1 2026-03-31T21:02:45.981 INFO:teuthology.orchestra.run.vm00.stdout:111669149703 2026-03-31T21:02:46.003 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149704 got 111669149703 for osd.3 2026-03-31T21:02:46.024 INFO:teuthology.orchestra.run.vm00.stdout:38654705692 2026-03-31T21:02:46.038 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705692 for osd.2 2026-03-31T21:02:46.065 INFO:teuthology.orchestra.run.vm00.stdout:38654705692 2026-03-31T21:02:46.082 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705692 for osd.0 2026-03-31T21:02:46.978 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:47.004 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:47.039 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:47.083 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:47.286 INFO:teuthology.orchestra.run.vm00.stdout:90194313237 2026-03-31T21:02:47.292 INFO:teuthology.orchestra.run.vm00.stdout:111669149704 2026-03-31T21:02:47.314 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149704 got 111669149704 for osd.3 2026-03-31T21:02:47.314 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:47.314 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313237 got 90194313237 for osd.1 2026-03-31T21:02:47.315 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:47.347 INFO:teuthology.orchestra.run.vm00.stdout:38654705693 2026-03-31T21:02:47.366 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705693 for osd.2 2026-03-31T21:02:47.366 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:47.371 INFO:teuthology.orchestra.run.vm00.stdout:38654705693 2026-03-31T21:02:47.389 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705693 for osd.0 2026-03-31T21:02:47.389 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:47.389 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:02:47.635 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:02:47.635 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:02:47.647 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2026-03-31T21:02:46.395660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6047,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7324,"num_bytes_recovered":60294988,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":275236,"kb_used_data":137652,"kb_used_omap":43,"kb_used_meta":137492,"kb_avail":282840284,"statfs":{"total":289910292480,"available":289628450816,"internally_reserved":0,"allocated":140955648,"data_stored":140451793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44836,"internal_metadata":140792028},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3420,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":28016640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.594235"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":28096,"reported_epoch":31,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:02:45.603314+0000","last_change":"2026-03-31T21:02:27.877871+0000","last_active":"2026-03-31T21:02:45.603314+0000","last_peered":"2026-03-31T21:02:45.603314+0000","last_clean":"2026-03-31T21:02:24.201524+0000","last_became_active":"2026-03-31T21:02:27.876634+0000","last_became_peered":"2026-03-31T21:02:27.876634+0000","last_unstale":"2026-03-31T21:02:45.603314+0000","last_undegraded":"2026-03-31T21:02:45.603314+0000","last_fullsized":"2026-03-31T21:02:45.603314+0000","mapping_epoch":29,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7322,"num_bytes_recovered":59966796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-03-31T21:02:45.392956+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:02:45.392956+0000","last_peered":"2026-03-31T21:02:45.392956+0000","last_clean":"2026-03-31T21:02:45.392956+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:02:45.392956+0000","last_undegraded":"2026-03-31T21:02:45.392956+0000","last_fullsized":"2026-03-31T21:02:45.392956+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7322,"num_bytes_recovered":59966796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194297856,"data_stored":194297856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83492,"kb_used_data":53716,"kb_used_omap":8,"kb_used_meta":29751,"kb_avail":94288348,"statfs":{"total":96636764160,"available":96551268352,"internally_reserved":0,"allocated":55005184,"data_stored":54841202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8381,"internal_metadata":30465859},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98920,"kb_used_data":54040,"kb_used_omap":17,"kb_used_meta":44846,"kb_avail":94272920,"statfs":{"total":96636764160,"available":96535470080,"internally_reserved":0,"allocated":55336960,"data_stored":55169394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17902,"internal_metadata":45922834},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313237,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705693,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92824,"kb_used_data":29896,"kb_used_omap":18,"kb_used_meta":62893,"kb_avail":94279016,"statfs":{"total":96636764160,"available":96541712384,"internally_reserved":0,"allocated":30613504,"data_stored":30441197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18553,"internal_metadata":64403335},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30052352,"data_stored":30052352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:02:47.647 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 28096, 'reported_epoch': 31, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:02:45.603314+0000', 'last_change': '2026-03-31T21:02:27.877871+0000', 'last_active': '2026-03-31T21:02:45.603314+0000', 'last_peered': '2026-03-31T21:02:45.603314+0000', 'last_clean': '2026-03-31T21:02:24.201524+0000', 'last_became_active': '2026-03-31T21:02:27.876634+0000', 'last_became_peered': '2026-03-31T21:02:27.876634+0000', 'last_unstale': '2026-03-31T21:02:45.603314+0000', 'last_undegraded': '2026-03-31T21:02:45.603314+0000', 'last_fullsized': '2026-03-31T21:02:45.603314+0000', 'mapping_epoch': 29, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 27, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6047, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 7322, 'num_bytes_recovered': 59966796, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:02:47.647 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:02:52.649 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:02:52.649 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:02:52.649 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:02:52.650 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:02:52.766 INFO:teuthology.orchestra.run.vm00.stdout:111669149707 2026-03-31T21:02:52.767 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:52.773 INFO:teuthology.orchestra.run.vm00.stdout:90194313240 2026-03-31T21:02:52.773 INFO:teuthology.orchestra.run.vm00.stdout:38654705695 2026-03-31T21:02:52.773 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:52.773 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:52.781 INFO:teuthology.orchestra.run.vm00.stdout:38654705695 2026-03-31T21:02:52.781 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:53.001 INFO:teuthology.orchestra.run.vm00.stdout:111669149705 2026-03-31T21:02:53.015 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149707 got 111669149705 for osd.3 2026-03-31T21:02:53.074 INFO:teuthology.orchestra.run.vm00.stdout:90194313239 2026-03-31T21:02:53.087 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313240 got 90194313239 for osd.1 2026-03-31T21:02:53.088 INFO:teuthology.orchestra.run.vm00.stdout:38654705694 2026-03-31T21:02:53.102 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705695 got 38654705694 for osd.2 2026-03-31T21:02:53.116 INFO:teuthology.orchestra.run.vm00.stdout:38654705694 2026-03-31T21:02:53.129 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705695 got 38654705694 for osd.0 2026-03-31T21:02:54.016 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:54.088 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:02:54.103 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:54.129 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:54.231 INFO:teuthology.orchestra.run.vm00.stdout:111669149707 2026-03-31T21:02:54.257 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149707 got 111669149707 for osd.3 2026-03-31T21:02:54.257 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:54.325 INFO:teuthology.orchestra.run.vm00.stdout:90194313240 2026-03-31T21:02:54.337 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313240 got 90194313240 for osd.1 2026-03-31T21:02:54.337 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:54.357 INFO:teuthology.orchestra.run.vm00.stdout:38654705696 2026-03-31T21:02:54.369 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705695 got 38654705696 for osd.2 2026-03-31T21:02:54.369 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:54.373 INFO:teuthology.orchestra.run.vm00.stdout:38654705696 2026-03-31T21:02:54.385 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705695 got 38654705696 for osd.0 2026-03-31T21:02:54.385 DEBUG:teuthology.parallel:result is None 2026-03-31T21:02:54.385 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:02:54.636 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:02:54.636 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:02:54.648 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2026-03-31T21:02:53.803470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4847,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8524,"num_bytes_recovered":70125388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":280044,"kb_used_data":142460,"kb_used_omap":46,"kb_used_meta":137489,"kb_avail":282835476,"statfs":{"total":289910292480,"available":289623527424,"internally_reserved":0,"allocated":145879040,"data_stored":145351630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48093,"internal_metadata":140788771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2456,"num_bytes_recovered":20119552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.414378"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":30498,"reported_epoch":32,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:02:48.315120+0000","last_change":"2026-03-31T21:02:27.877871+0000","last_active":"2026-03-31T21:02:48.315120+0000","last_peered":"2026-03-31T21:02:48.315120+0000","last_clean":"2026-03-31T21:02:24.201524+0000","last_became_active":"2026-03-31T21:02:27.876634+0000","last_became_peered":"2026-03-31T21:02:27.876634+0000","last_unstale":"2026-03-31T21:02:48.315120+0000","last_undegraded":"2026-03-31T21:02:48.315120+0000","last_fullsized":"2026-03-31T21:02:48.315120+0000","mapping_epoch":29,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4847,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8522,"num_bytes_recovered":69797196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":61,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-03-31T21:02:46.398354+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:02:46.398354+0000","last_peered":"2026-03-31T21:02:46.398354+0000","last_clean":"2026-03-31T21:02:46.398354+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:02:46.398354+0000","last_undegraded":"2026-03-31T21:02:46.398354+0000","last_fullsized":"2026-03-31T21:02:46.398354+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4847,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8522,"num_bytes_recovered":69797196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199147520,"data_stored":199147520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149707,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83508,"kb_used_data":53732,"kb_used_omap":9,"kb_used_meta":29750,"kb_avail":94288332,"statfs":{"total":96636764160,"available":96551251968,"internally_reserved":0,"allocated":55021568,"data_stored":54848154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9685,"internal_metadata":30464555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98936,"kb_used_data":54056,"kb_used_omap":18,"kb_used_meta":44845,"kb_avail":94272904,"statfs":{"total":96636764160,"available":96535453696,"internally_reserved":0,"allocated":55353344,"data_stored":55176346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":45921532},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313240,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705696,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97600,"kb_used_data":34672,"kb_used_omap":18,"kb_used_meta":62893,"kb_avail":94274240,"statfs":{"total":96636764160,"available":96536821760,"internally_reserved":0,"allocated":35504128,"data_stored":35327130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":64402684},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34902016,"data_stored":34902016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:02:54.648 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 30498, 'reported_epoch': 32, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:02:48.315120+0000', 'last_change': '2026-03-31T21:02:27.877871+0000', 'last_active': '2026-03-31T21:02:48.315120+0000', 'last_peered': '2026-03-31T21:02:48.315120+0000', 'last_clean': '2026-03-31T21:02:24.201524+0000', 'last_became_active': '2026-03-31T21:02:27.876634+0000', 'last_became_peered': '2026-03-31T21:02:27.876634+0000', 'last_unstale': '2026-03-31T21:02:48.315120+0000', 'last_undegraded': '2026-03-31T21:02:48.315120+0000', 'last_fullsized': '2026-03-31T21:02:48.315120+0000', 'mapping_epoch': 29, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 27, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4847, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8522, 'num_bytes_recovered': 69797196, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:02:54.648 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:02:59.650 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:02:59.650 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:02:59.650 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:02:59.650 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:02:59.767 INFO:teuthology.orchestra.run.vm00.stdout:38654705698 2026-03-31T21:02:59.767 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:02:59.770 INFO:teuthology.orchestra.run.vm00.stdout:111669149709 2026-03-31T21:02:59.770 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:02:59.770 INFO:teuthology.orchestra.run.vm00.stdout:38654705698 2026-03-31T21:02:59.770 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:02:59.771 INFO:teuthology.orchestra.run.vm00.stdout:90194313242 2026-03-31T21:02:59.771 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:00.002 INFO:teuthology.orchestra.run.vm00.stdout:38654705698 2026-03-31T21:03:00.018 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705698 got 38654705698 for osd.2 2026-03-31T21:03:00.018 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:00.033 INFO:teuthology.orchestra.run.vm00.stdout:38654705698 2026-03-31T21:03:00.036 INFO:teuthology.orchestra.run.vm00.stdout:90194313242 2026-03-31T21:03:00.046 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705698 got 38654705698 for osd.0 2026-03-31T21:03:00.046 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:00.047 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313242 got 90194313242 for osd.1 2026-03-31T21:03:00.047 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:00.058 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:03:00.056+0000 7f0339864640 0 --2- 192.168.123.100:0/2122676643 >> v2:192.168.123.100:3300/0 conn(0x7f033407f830 0x7f033407fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:03:00.189 INFO:teuthology.orchestra.run.vm00.stdout:111669149709 2026-03-31T21:03:00.201 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149709 got 111669149709 for osd.3 2026-03-31T21:03:00.201 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:00.201 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:00.513 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:03:00.511+0000 7f99b51d0640 0 --2- 192.168.123.100:0/1443559960 >> [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] conn(0x7f99b007f830 0x7f99b0173b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:03:00.639 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:00.639 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:00.650 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2026-03-31T21:02:59.804312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4847,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8524,"num_bytes_recovered":70125388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":280044,"kb_used_data":142460,"kb_used_omap":46,"kb_used_meta":137489,"kb_avail":282835476,"statfs":{"total":289910292480,"available":289623527424,"internally_reserved":0,"allocated":145879040,"data_stored":145351630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48093,"internal_metadata":140788771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-364,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":364,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001738"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":30498,"reported_epoch":32,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:02:48.315120+0000","last_change":"2026-03-31T21:02:27.877871+0000","last_active":"2026-03-31T21:02:48.315120+0000","last_peered":"2026-03-31T21:02:48.315120+0000","last_clean":"2026-03-31T21:02:24.201524+0000","last_became_active":"2026-03-31T21:02:27.876634+0000","last_became_peered":"2026-03-31T21:02:27.876634+0000","last_unstale":"2026-03-31T21:02:48.315120+0000","last_undegraded":"2026-03-31T21:02:48.315120+0000","last_fullsized":"2026-03-31T21:02:48.315120+0000","mapping_epoch":29,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4847,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8522,"num_bytes_recovered":69797196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":61,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-03-31T21:02:46.398354+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:02:46.398354+0000","last_peered":"2026-03-31T21:02:46.398354+0000","last_clean":"2026-03-31T21:02:46.398354+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:02:46.398354+0000","last_undegraded":"2026-03-31T21:02:46.398354+0000","last_fullsized":"2026-03-31T21:02:46.398354+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4847,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8522,"num_bytes_recovered":69797196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199147520,"data_stored":199147520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83508,"kb_used_data":53732,"kb_used_omap":9,"kb_used_meta":29750,"kb_avail":94288332,"statfs":{"total":96636764160,"available":96551251968,"internally_reserved":0,"allocated":55021568,"data_stored":54848154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9685,"internal_metadata":30464555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98936,"kb_used_data":54056,"kb_used_omap":18,"kb_used_meta":44845,"kb_avail":94272904,"statfs":{"total":96636764160,"available":96535453696,"internally_reserved":0,"allocated":55353344,"data_stored":55176346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":45921532},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313242,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705698,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97600,"kb_used_data":34672,"kb_used_omap":18,"kb_used_meta":62893,"kb_avail":94274240,"statfs":{"total":96636764160,"available":96536821760,"internally_reserved":0,"allocated":35504128,"data_stored":35327130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":64402684},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34902016,"data_stored":34902016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:00.650 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 30498, 'reported_epoch': 32, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:02:48.315120+0000', 'last_change': '2026-03-31T21:02:27.877871+0000', 'last_active': '2026-03-31T21:02:48.315120+0000', 'last_peered': '2026-03-31T21:02:48.315120+0000', 'last_clean': '2026-03-31T21:02:24.201524+0000', 'last_became_active': '2026-03-31T21:02:27.876634+0000', 'last_became_peered': '2026-03-31T21:02:27.876634+0000', 'last_unstale': '2026-03-31T21:02:48.315120+0000', 'last_undegraded': '2026-03-31T21:02:48.315120+0000', 'last_fullsized': '2026-03-31T21:02:48.315120+0000', 'mapping_epoch': 29, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 27, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4847, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 8522, 'num_bytes_recovered': 69797196, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:00.651 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:03:04.341 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:03:04.339+0000 7f4f15fc9640 -1 osd.3 pg_epoch: 32 pg[2.0s0( v 22'13367 (22'13341,22'13367] local-lis/les=29/30 n=13367 ec=15/15 lis/c=29/26 les/c/f=30/27/0 sis=29) [3,0,2]/[3,1,2]p3(0) backfill=[0(1)] r=0 lpr=29 pi=[26,29)/1 crt=22'13367 lcod 0'0 mlcod 0'0 active+remapped+backfilling rops=1 mbc={0={},1={},2={}}] ECBackendL continue_recovery_op: 2:ad551702:::test:head has inconsistent hinfo 2026-03-31T21:03:04.341 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:03:04.339+0000 7f4f15fc9640 -1 log_channel(cluster) log [ERR] : 2.0s0 missing primary copy of 2:ad551702:::test:head, will try copies on 1(1),2(2) 2026-03-31T21:03:05.651 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:03:05.651 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:03:05.651 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:03:05.652 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:03:05.769 INFO:teuthology.orchestra.run.vm00.stdout:90194313244 2026-03-31T21:03:05.769 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:05.772 INFO:teuthology.orchestra.run.vm00.stdout:38654705700 2026-03-31T21:03:05.773 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:05.778 INFO:teuthology.orchestra.run.vm00.stdout:38654705700 2026-03-31T21:03:05.778 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:05.778 INFO:teuthology.orchestra.run.vm00.stdout:111669149711 2026-03-31T21:03:05.778 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:06.115 INFO:teuthology.orchestra.run.vm00.stdout:38654705700 2026-03-31T21:03:06.117 INFO:teuthology.orchestra.run.vm00.stdout:90194313244 2026-03-31T21:03:06.118 INFO:teuthology.orchestra.run.vm00.stdout:38654705700 2026-03-31T21:03:06.131 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313244 got 90194313244 for osd.1 2026-03-31T21:03:06.131 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:06.131 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705700 got 38654705700 for osd.2 2026-03-31T21:03:06.131 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:06.133 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705700 got 38654705700 for osd.0 2026-03-31T21:03:06.133 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:06.148 INFO:teuthology.orchestra.run.vm00.stdout:111669149711 2026-03-31T21:03:06.160 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149711 got 111669149711 for osd.3 2026-03-31T21:03:06.160 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:06.160 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:06.463 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:06.463 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:06.479 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2026-03-31T21:03:05.805303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3725,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9646,"num_bytes_recovered":79301130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":284468,"kb_used_data":146884,"kb_used_omap":46,"kb_used_meta":137489,"kb_avail":282831052,"statfs":{"total":289910292480,"available":289618997248,"internally_reserved":0,"allocated":150409216,"data_stored":149881806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48093,"internal_metadata":140788771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":9175742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001833"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":32743,"reported_epoch":32,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:03:05.667606+0000","last_change":"2026-03-31T21:02:27.877871+0000","last_active":"2026-03-31T21:03:05.667606+0000","last_peered":"2026-03-31T21:03:05.667606+0000","last_clean":"2026-03-31T21:02:24.201524+0000","last_became_active":"2026-03-31T21:02:27.876634+0000","last_became_peered":"2026-03-31T21:02:27.876634+0000","last_unstale":"2026-03-31T21:03:05.667606+0000","last_undegraded":"2026-03-31T21:03:05.667606+0000","last_fullsized":"2026-03-31T21:03:05.667606+0000","mapping_epoch":29,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3725,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9644,"num_bytes_recovered":78972938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":61,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-03-31T21:02:46.398354+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:02:46.398354+0000","last_peered":"2026-03-31T21:02:46.398354+0000","last_clean":"2026-03-31T21:02:46.398354+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:02:46.398354+0000","last_undegraded":"2026-03-31T21:02:46.398354+0000","last_fullsized":"2026-03-31T21:02:46.398354+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3725,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9644,"num_bytes_recovered":78972938,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203747328,"data_stored":203747328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83508,"kb_used_data":53732,"kb_used_omap":9,"kb_used_meta":29750,"kb_avail":94288332,"statfs":{"total":96636764160,"available":96551251968,"internally_reserved":0,"allocated":55021568,"data_stored":54848154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9685,"internal_metadata":30464555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98936,"kb_used_data":54056,"kb_used_omap":18,"kb_used_meta":44845,"kb_avail":94272904,"statfs":{"total":96636764160,"available":96535453696,"internally_reserved":0,"allocated":55353344,"data_stored":55176346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":45921532},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313244,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102024,"kb_used_data":39096,"kb_used_omap":18,"kb_used_meta":62893,"kb_avail":94269816,"statfs":{"total":96636764160,"available":96532291584,"internally_reserved":0,"allocated":40034304,"data_stored":39857306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":64402684},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39501824,"data_stored":39501824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:06.479 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 32743, 'reported_epoch': 32, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:03:05.667606+0000', 'last_change': '2026-03-31T21:02:27.877871+0000', 'last_active': '2026-03-31T21:03:05.667606+0000', 'last_peered': '2026-03-31T21:03:05.667606+0000', 'last_clean': '2026-03-31T21:02:24.201524+0000', 'last_became_active': '2026-03-31T21:02:27.876634+0000', 'last_became_peered': '2026-03-31T21:02:27.876634+0000', 'last_unstale': '2026-03-31T21:03:05.667606+0000', 'last_undegraded': '2026-03-31T21:03:05.667606+0000', 'last_fullsized': '2026-03-31T21:03:05.667606+0000', 'mapping_epoch': 29, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 27, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3725, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 9644, 'num_bytes_recovered': 78972938, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:06.479 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:03:11.480 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:03:11.480 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:03:11.480 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:03:11.480 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:03:11.639 INFO:teuthology.orchestra.run.vm00.stdout:111669149713 2026-03-31T21:03:11.639 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:11.640 INFO:teuthology.orchestra.run.vm00.stdout:90194313246 2026-03-31T21:03:11.640 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:11.640 INFO:teuthology.orchestra.run.vm00.stdout:38654705702 2026-03-31T21:03:11.641 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:11.697 INFO:teuthology.orchestra.run.vm00.stdout:38654705702 2026-03-31T21:03:11.697 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:11.954 INFO:teuthology.orchestra.run.vm00.stdout:111669149712 2026-03-31T21:03:11.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149713 got 111669149712 for osd.3 2026-03-31T21:03:12.021 INFO:teuthology.orchestra.run.vm00.stdout:90194313247 2026-03-31T21:03:12.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313246 got 90194313247 for osd.1 2026-03-31T21:03:12.043 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:12.043 INFO:teuthology.orchestra.run.vm00.stdout:38654705702 2026-03-31T21:03:12.057 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705702 for osd.0 2026-03-31T21:03:12.058 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:12.059 INFO:teuthology.orchestra.run.vm00.stdout:38654705702 2026-03-31T21:03:12.072 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705702 for osd.2 2026-03-31T21:03:12.072 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:12.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:13.244 INFO:teuthology.orchestra.run.vm00.stdout:111669149713 2026-03-31T21:03:13.258 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149713 got 111669149713 for osd.3 2026-03-31T21:03:13.258 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:13.258 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:13.490 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:13.490 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:13.502 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2026-03-31T21:03:11.806070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":932,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12439,"num_bytes_recovered":102181386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":293932,"kb_used_data":156348,"kb_used_omap":46,"kb_used_meta":137489,"kb_avail":282821588,"statfs":{"total":289910292480,"available":289609306112,"internally_reserved":0,"allocated":160100352,"data_stored":159572942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48093,"internal_metadata":140788771},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3915,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3915,"num_bytes_recovered":32055998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001757"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":38329,"reported_epoch":32,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:03:11.619358+0000","last_change":"2026-03-31T21:02:27.877871+0000","last_active":"2026-03-31T21:03:11.619358+0000","last_peered":"2026-03-31T21:03:11.619358+0000","last_clean":"2026-03-31T21:02:24.201524+0000","last_became_active":"2026-03-31T21:02:27.876634+0000","last_became_peered":"2026-03-31T21:02:27.876634+0000","last_unstale":"2026-03-31T21:03:11.619358+0000","last_undegraded":"2026-03-31T21:03:11.619358+0000","last_fullsized":"2026-03-31T21:03:11.619358+0000","mapping_epoch":29,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":932,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12437,"num_bytes_recovered":101853194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":61,"reported_epoch":32,"state":"active+clean","last_fresh":"2026-03-31T21:02:46.398354+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:02:46.398354+0000","last_peered":"2026-03-31T21:02:46.398354+0000","last_clean":"2026-03-31T21:02:46.398354+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:02:46.398354+0000","last_undegraded":"2026-03-31T21:02:46.398354+0000","last_fullsized":"2026-03-31T21:02:46.398354+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":932,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12437,"num_bytes_recovered":101853194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215195648,"data_stored":215195648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83508,"kb_used_data":53732,"kb_used_omap":9,"kb_used_meta":29750,"kb_avail":94288332,"statfs":{"total":96636764160,"available":96551251968,"internally_reserved":0,"allocated":55021568,"data_stored":54848154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9685,"internal_metadata":30464555},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98936,"kb_used_data":54056,"kb_used_omap":18,"kb_used_meta":44845,"kb_avail":94272904,"statfs":{"total":96636764160,"available":96535453696,"internally_reserved":0,"allocated":55353344,"data_stored":55176346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":45921532},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313247,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111488,"kb_used_data":48560,"kb_used_omap":18,"kb_used_meta":62893,"kb_avail":94260352,"statfs":{"total":96636764160,"available":96522600448,"internally_reserved":0,"allocated":49725440,"data_stored":49548442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19204,"internal_metadata":64402684},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50950144,"data_stored":50950144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:13.502 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 38329, 'reported_epoch': 32, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:03:11.619358+0000', 'last_change': '2026-03-31T21:02:27.877871+0000', 'last_active': '2026-03-31T21:03:11.619358+0000', 'last_peered': '2026-03-31T21:03:11.619358+0000', 'last_clean': '2026-03-31T21:02:24.201524+0000', 'last_became_active': '2026-03-31T21:02:27.876634+0000', 'last_became_peered': '2026-03-31T21:02:27.876634+0000', 'last_unstale': '2026-03-31T21:03:11.619358+0000', 'last_undegraded': '2026-03-31T21:03:11.619358+0000', 'last_fullsized': '2026-03-31T21:03:11.619358+0000', 'mapping_epoch': 29, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 27, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 932, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12437, 'num_bytes_recovered': 101853194, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:13.502 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:03:18.503 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:03:18.503 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:03:18.503 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:03:18.503 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:03:18.618 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:18.618 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:18.637 INFO:teuthology.orchestra.run.vm00.stdout:90194313249 2026-03-31T21:03:18.637 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:18.638 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:18.638 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:18.656 INFO:teuthology.orchestra.run.vm00.stdout:111669149716 2026-03-31T21:03:18.656 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:18.859 INFO:teuthology.orchestra.run.vm00.stdout:38654705703 2026-03-31T21:03:18.867 INFO:teuthology.orchestra.run.vm00.stdout:38654705703 2026-03-31T21:03:18.877 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705703 for osd.2 2026-03-31T21:03:18.880 INFO:teuthology.orchestra.run.vm00.stdout:90194313248 2026-03-31T21:03:18.881 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705703 for osd.0 2026-03-31T21:03:18.892 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313249 got 90194313248 for osd.1 2026-03-31T21:03:19.357 INFO:teuthology.orchestra.run.vm00.stdout:111669149715 2026-03-31T21:03:19.369 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149716 got 111669149715 for osd.3 2026-03-31T21:03:19.878 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:19.882 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:19.893 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:20.092 INFO:teuthology.orchestra.run.vm00.stdout:38654705703 2026-03-31T21:03:20.106 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705703 for osd.2 2026-03-31T21:03:20.142 INFO:teuthology.orchestra.run.vm00.stdout:90194313248 2026-03-31T21:03:20.145 INFO:teuthology.orchestra.run.vm00.stdout:38654705703 2026-03-31T21:03:20.156 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313249 got 90194313248 for osd.1 2026-03-31T21:03:20.163 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705703 for osd.0 2026-03-31T21:03:20.369 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:20.576 INFO:teuthology.orchestra.run.vm00.stdout:111669149716 2026-03-31T21:03:20.588 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149716 got 111669149716 for osd.3 2026-03-31T21:03:20.589 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:21.107 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:21.157 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:21.164 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:21.323 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:21.338 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705705 for osd.2 2026-03-31T21:03:21.338 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:21.374 INFO:teuthology.orchestra.run.vm00.stdout:90194313249 2026-03-31T21:03:21.388 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313249 got 90194313249 for osd.1 2026-03-31T21:03:21.388 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:21.410 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:21.421 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705705 got 38654705705 for osd.0 2026-03-31T21:03:21.421 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:21.421 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:21.658 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:21.658 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:21.670 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2026-03-31T21:03:19.807228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13370,"num_bytes_recovered":109808138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":299476,"kb_used_data":161892,"kb_used_omap":50,"kb_used_meta":137485,"kb_avail":282816044,"statfs":{"total":289910292480,"available":289603629056,"internally_reserved":0,"allocated":165777408,"data_stored":165221165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51350,"internal_metadata":140785514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-932,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":7626752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001157"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40206,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.111251+0000","last_change":"2026-03-31T21:03:15.111192+0000","last_active":"2026-03-31T21:03:15.111251+0000","last_peered":"2026-03-31T21:03:15.111251+0000","last_clean":"2026-03-31T21:03:15.111251+0000","last_became_active":"2026-03-31T21:03:15.110760+0000","last_became_peered":"2026-03-31T21:03:15.110760+0000","last_unstale":"2026-03-31T21:03:15.111251+0000","last_undegraded":"2026-03-31T21:03:15.111251+0000","last_fullsized":"2026-03-31T21:03:15.111251+0000","mapping_epoch":33,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T23:16:15.167921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":65,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.108214+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:03:15.108214+0000","last_peered":"2026-03-31T21:03:15.108214+0000","last_clean":"2026-03-31T21:03:15.108214+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:03:15.108214+0000","last_undegraded":"2026-03-31T21:03:15.108214+0000","last_fullsized":"2026-03-31T21:03:15.108214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218877952,"data_stored":218877952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83524,"kb_used_data":53748,"kb_used_omap":10,"kb_used_meta":29749,"kb_avail":94288316,"statfs":{"total":96636764160,"available":96551235584,"internally_reserved":0,"allocated":55037952,"data_stored":54854927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10989,"internal_metadata":30463251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98952,"kb_used_data":54072,"kb_used_omap":20,"kb_used_meta":44843,"kb_avail":94272888,"statfs":{"total":96636764160,"available":96535437312,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20506,"internal_metadata":45920230},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313249,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117000,"kb_used_data":54072,"kb_used_omap":19,"kb_used_meta":62892,"kb_avail":94254840,"statfs":{"total":96636764160,"available":96516956160,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19855,"internal_metadata":64402033},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54624256,"data_stored":54624256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:21.670 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40206, 'reported_epoch': 34, 'state': 'active+clean', 'last_fresh': '2026-03-31T21:03:15.111251+0000', 'last_change': '2026-03-31T21:03:15.111192+0000', 'last_active': '2026-03-31T21:03:15.111251+0000', 'last_peered': '2026-03-31T21:03:15.111251+0000', 'last_clean': '2026-03-31T21:03:15.111251+0000', 'last_became_active': '2026-03-31T21:03:15.110760+0000', 'last_became_peered': '2026-03-31T21:03:15.110760+0000', 'last_unstale': '2026-03-31T21:03:15.111251+0000', 'last_undegraded': '2026-03-31T21:03:15.111251+0000', 'last_fullsized': '2026-03-31T21:03:15.111251+0000', 'mapping_epoch': 33, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 34, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-04-01T23:16:15.167921+0000', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13368, 'num_bytes_recovered': 109479946, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:21.670 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-03-31T21:03:21.670 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-03-31T21:03:21.670 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:21.949 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:21.949 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:21.962 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2026-03-31T21:03:21.807489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13370,"num_bytes_recovered":109808138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":299476,"kb_used_data":161892,"kb_used_omap":50,"kb_used_meta":137485,"kb_avail":282816044,"statfs":{"total":289910292480,"available":289603629056,"internally_reserved":0,"allocated":165777408,"data_stored":165221165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51350,"internal_metadata":140785514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-631,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001148"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40206,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.111251+0000","last_change":"2026-03-31T21:03:15.111192+0000","last_active":"2026-03-31T21:03:15.111251+0000","last_peered":"2026-03-31T21:03:15.111251+0000","last_clean":"2026-03-31T21:03:15.111251+0000","last_became_active":"2026-03-31T21:03:15.110760+0000","last_became_peered":"2026-03-31T21:03:15.110760+0000","last_unstale":"2026-03-31T21:03:15.111251+0000","last_undegraded":"2026-03-31T21:03:15.111251+0000","last_fullsized":"2026-03-31T21:03:15.111251+0000","mapping_epoch":33,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T23:16:15.167921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":65,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.108214+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:03:15.108214+0000","last_peered":"2026-03-31T21:03:15.108214+0000","last_clean":"2026-03-31T21:03:15.108214+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:03:15.108214+0000","last_undegraded":"2026-03-31T21:03:15.108214+0000","last_fullsized":"2026-03-31T21:03:15.108214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218755072,"data_stored":218755072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83524,"kb_used_data":53748,"kb_used_omap":10,"kb_used_meta":29749,"kb_avail":94288316,"statfs":{"total":96636764160,"available":96551235584,"internally_reserved":0,"allocated":55037952,"data_stored":54854927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10989,"internal_metadata":30463251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98952,"kb_used_data":54072,"kb_used_omap":20,"kb_used_meta":44843,"kb_avail":94272888,"statfs":{"total":96636764160,"available":96535437312,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20506,"internal_metadata":45920230},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313250,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117000,"kb_used_data":54072,"kb_used_omap":19,"kb_used_meta":62892,"kb_avail":94254840,"statfs":{"total":96636764160,"available":96516956160,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19855,"internal_metadata":64402033},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54501376,"data_stored":54501376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:21.962 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:22.154 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:22.154 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:22.165 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2026-03-31T21:03:21.807489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13370,"num_bytes_recovered":109808138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":299476,"kb_used_data":161892,"kb_used_omap":50,"kb_used_meta":137485,"kb_avail":282816044,"statfs":{"total":289910292480,"available":289603629056,"internally_reserved":0,"allocated":165777408,"data_stored":165221165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51350,"internal_metadata":140785514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-631,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001148"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40206,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.111251+0000","last_change":"2026-03-31T21:03:15.111192+0000","last_active":"2026-03-31T21:03:15.111251+0000","last_peered":"2026-03-31T21:03:15.111251+0000","last_clean":"2026-03-31T21:03:15.111251+0000","last_became_active":"2026-03-31T21:03:15.110760+0000","last_became_peered":"2026-03-31T21:03:15.110760+0000","last_unstale":"2026-03-31T21:03:15.111251+0000","last_undegraded":"2026-03-31T21:03:15.111251+0000","last_fullsized":"2026-03-31T21:03:15.111251+0000","mapping_epoch":33,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T23:16:15.167921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":65,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.108214+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:03:15.108214+0000","last_peered":"2026-03-31T21:03:15.108214+0000","last_clean":"2026-03-31T21:03:15.108214+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:03:15.108214+0000","last_undegraded":"2026-03-31T21:03:15.108214+0000","last_fullsized":"2026-03-31T21:03:15.108214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218755072,"data_stored":218755072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83524,"kb_used_data":53748,"kb_used_omap":10,"kb_used_meta":29749,"kb_avail":94288316,"statfs":{"total":96636764160,"available":96551235584,"internally_reserved":0,"allocated":55037952,"data_stored":54854927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10989,"internal_metadata":30463251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98952,"kb_used_data":54072,"kb_used_omap":20,"kb_used_meta":44843,"kb_avail":94272888,"statfs":{"total":96636764160,"available":96535437312,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20506,"internal_metadata":45920230},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313250,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117000,"kb_used_data":54072,"kb_used_omap":19,"kb_used_meta":62892,"kb_avail":94254840,"statfs":{"total":96636764160,"available":96516956160,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19855,"internal_metadata":64402033},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54501376,"data_stored":54501376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:22.165 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-03-31T21:03:22.166 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:03:22.166 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:03:22.166 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:03:22.166 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:03:22.290 INFO:teuthology.orchestra.run.vm00.stdout:90194313251 2026-03-31T21:03:22.290 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:22.291 INFO:teuthology.orchestra.run.vm00.stdout:38654705706 2026-03-31T21:03:22.291 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:22.298 INFO:teuthology.orchestra.run.vm00.stdout:38654705706 2026-03-31T21:03:22.298 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:22.302 INFO:teuthology.orchestra.run.vm00.stdout:111669149718 2026-03-31T21:03:22.302 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:22.520 INFO:teuthology.orchestra.run.vm00.stdout:90194313250 2026-03-31T21:03:22.527 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:22.527 INFO:teuthology.orchestra.run.vm00.stdout:111669149716 2026-03-31T21:03:22.530 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:22.536 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313251 got 90194313250 for osd.1 2026-03-31T21:03:22.543 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705705 for osd.2 2026-03-31T21:03:22.543 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149718 got 111669149716 for osd.3 2026-03-31T21:03:22.544 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705705 for osd.0 2026-03-31T21:03:23.537 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:23.543 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:23.543 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:23.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:23.784 INFO:teuthology.orchestra.run.vm00.stdout:90194313250 2026-03-31T21:03:23.801 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313251 got 90194313250 for osd.1 2026-03-31T21:03:23.801 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:23.803 INFO:teuthology.orchestra.run.vm00.stdout:38654705705 2026-03-31T21:03:23.812 INFO:teuthology.orchestra.run.vm00.stdout:111669149716 2026-03-31T21:03:23.816 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705705 for osd.2 2026-03-31T21:03:23.819 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705705 for osd.0 2026-03-31T21:03:23.825 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149718 got 111669149716 for osd.3 2026-03-31T21:03:24.801 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:24.817 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:24.819 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:24.826 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:25.039 INFO:teuthology.orchestra.run.vm00.stdout:90194313251 2026-03-31T21:03:25.060 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313251 got 90194313251 for osd.1 2026-03-31T21:03:25.060 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:25.069 INFO:teuthology.orchestra.run.vm00.stdout:38654705707 2026-03-31T21:03:25.071 INFO:teuthology.orchestra.run.vm00.stdout:111669149718 2026-03-31T21:03:25.076 INFO:teuthology.orchestra.run.vm00.stdout:38654705707 2026-03-31T21:03:25.087 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705707 for osd.2 2026-03-31T21:03:25.087 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:25.088 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705707 for osd.0 2026-03-31T21:03:25.088 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:25.090 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149718 got 111669149718 for osd.3 2026-03-31T21:03:25.090 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:25.090 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:25.328 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:25.328 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:25.340 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2026-03-31T21:03:23.807724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13370,"num_bytes_recovered":109808138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":299476,"kb_used_data":161892,"kb_used_omap":50,"kb_used_meta":137485,"kb_avail":282816044,"statfs":{"total":289910292480,"available":289603629056,"internally_reserved":0,"allocated":165777408,"data_stored":165221165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51350,"internal_metadata":140785514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-631,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":630,"num_bytes_recovered":5160960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.723964"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40206,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.111251+0000","last_change":"2026-03-31T21:03:15.111192+0000","last_active":"2026-03-31T21:03:15.111251+0000","last_peered":"2026-03-31T21:03:15.111251+0000","last_clean":"2026-03-31T21:03:15.111251+0000","last_became_active":"2026-03-31T21:03:15.110760+0000","last_became_peered":"2026-03-31T21:03:15.110760+0000","last_unstale":"2026-03-31T21:03:15.111251+0000","last_undegraded":"2026-03-31T21:03:15.111251+0000","last_fullsized":"2026-03-31T21:03:15.111251+0000","mapping_epoch":33,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T23:16:15.167921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":65,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-03-31T21:03:15.108214+0000","last_change":"2026-03-31T21:02:27.171032+0000","last_active":"2026-03-31T21:03:15.108214+0000","last_peered":"2026-03-31T21:03:15.108214+0000","last_clean":"2026-03-31T21:03:15.108214+0000","last_became_active":"2026-03-31T21:02:26.865303+0000","last_became_peered":"2026-03-31T21:02:26.865303+0000","last_unstale":"2026-03-31T21:03:15.108214+0000","last_undegraded":"2026-03-31T21:03:15.108214+0000","last_fullsized":"2026-03-31T21:03:15.108214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T01:35:58.749780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13368,"num_bytes_recovered":109479946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218755072,"data_stored":218755072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":656384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83524,"kb_used_data":53748,"kb_used_omap":10,"kb_used_meta":29749,"kb_avail":94288316,"statfs":{"total":96636764160,"available":96551235584,"internally_reserved":0,"allocated":55037952,"data_stored":54854927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10989,"internal_metadata":30463251},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98952,"kb_used_data":54072,"kb_used_omap":20,"kb_used_meta":44843,"kb_avail":94272888,"statfs":{"total":96636764160,"available":96535437312,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20506,"internal_metadata":45920230},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":90194313251,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705707,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117000,"kb_used_data":54072,"kb_used_omap":19,"kb_used_meta":62892,"kb_avail":94254840,"statfs":{"total":96636764160,"available":96516956160,"internally_reserved":0,"allocated":55369728,"data_stored":55183119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19855,"internal_metadata":64402033},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54501376,"data_stored":54501376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:25.340 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40206, 'reported_epoch': 34, 'state': 'active+clean', 'last_fresh': '2026-03-31T21:03:15.111251+0000', 'last_change': '2026-03-31T21:03:15.111192+0000', 'last_active': '2026-03-31T21:03:15.111251+0000', 'last_peered': '2026-03-31T21:03:15.111251+0000', 'last_clean': '2026-03-31T21:03:15.111251+0000', 'last_became_active': '2026-03-31T21:03:15.110760+0000', 'last_became_peered': '2026-03-31T21:03:15.110760+0000', 'last_unstale': '2026-03-31T21:03:15.111251+0000', 'last_undegraded': '2026-03-31T21:03:15.111251+0000', 'last_fullsized': '2026-03-31T21:03:15.111251+0000', 'mapping_epoch': 33, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 34, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-04-01T23:16:15.167921+0000', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13368, 'num_bytes_recovered': 109479946, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:25.340 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-03-31T21:03:25.584 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:25.584 INFO:teuthology.orchestra.run.vm00.stdout:{"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":148,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":34,"num_osds":4,"num_up_osds":4,"osd_up_since":1774990943,"num_in_osds":3,"osd_in_since":1774990946,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":13369,"data_bytes":109807460,"bytes_used":306663424,"bytes_avail":289603629056,"bytes_total":289910292480,"recovering_objects_per_sec":64,"recovering_bytes_per_sec":530746,"recovering_keys_per_sec":0,"num_objects_recovered":630,"num_bytes_recovered":5160960,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2026-03-31T21:00:56:876132+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-03-31T21:03:23.807763+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-03-31T21:03:25.596 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'edf0be71-aa1c-40ef-819b-fbd038c1604f', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 4, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 148, 'monmap': {'epoch': 1, 'min_mon_release_name': 'tentacle', 'num_mons': 3}, 'osdmap': {'epoch': 34, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1774990943, 'num_in_osds': 3, 'osd_in_since': 1774990946, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 13369, 'data_bytes': 109807460, 'bytes_used': 306663424, 'bytes_avail': 289603629056, 'bytes_total': 289910292480, 'recovering_objects_per_sec': 64, 'recovering_bytes_per_sec': 530746, 'recovering_keys_per_sec': 0, 'num_objects_recovered': 630, 'num_bytes_recovered': 5160960, 'num_keys_recovered': 0}, 'fsmap': {'epoch': 1, 'btime': '2026-03-31T21:00:56:876132+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs'], 'services': {}}, 'servicemap': {'epoch': 5, 'modified': '2026-03-31T21:03:23.807763+0000', 'services': {'osd': {'daemons': {'summary': '', '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2026-03-31T21:03:25.596 DEBUG:tasks.ec_inconsistent_hinfo:there are 0 unfound objects 2026-03-31T21:03:25.596 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from non-primary osd.0 shard and test backfill 2026-03-31T21:03:25.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:03:25.803 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:25.803 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":34,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:03:15.097636+0000","last_up_change":"2026-03-31T21:02:23.181592+0000","last_in_change":"2026-03-31T21:02:26.197000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":2759845058}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":2759845058}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":2759845058}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":2759845058}]},"public_addr":"192.168.123.100:6812/2759845058","cluster_addr":"192.168.123.100:6813/2759845058","heartbeat_back_addr":"192.168.123.100:6815/2759845058","heartbeat_front_addr":"192.168.123.100:6814/2759845058","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":23,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":26,"up_thru":33,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":2551005687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":2551005687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":2551005687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":2551005687}]},"public_addr":"192.168.123.100:6804/2551005687","cluster_addr":"192.168.123.100:6805/2551005687","heartbeat_back_addr":"192.168.123.100:6807/2551005687","heartbeat_front_addr":"192.168.123.100:6806/2551005687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.978308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":65536,"last_purged_snaps_scrub":"2026-03-31T21:00:59.991163+0000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.868703+0000","dead_epoch":23}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:03:25.816 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-03-31T21:03:26.028 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:26.038 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":34,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,0,2],"up_primary":3,"acting":[3,0,2],"acting_primary":3} 2026-03-31T21:03:26.039 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-03-31T21:03:26.039 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:03:26.072 INFO:tasks.ceph.osd.0:Stopped 2026-03-31T21:03:26.072 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 "$object" rm-attr hinfo_key 2026-03-31T21:03:26.072 DEBUG:teuthology.orchestra.run.vm00:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list |grep '"'"'"oid":"test"'"'"') 2026-03-31T21:03:26.072 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 "$object" rm-attr hinfo_key' 2026-03-31T21:03:26.128 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -f /etc/bashrc ']' 2026-03-31T21:03:26.128 INFO:teuthology.orchestra.run.vm00.stderr:+ . /etc/bashrc 2026-03-31T21:03:26.128 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -z '' ']' 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ BASHRCSOURCED=Y 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ shopt -q login_shell 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:+++ umask 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' 0022 -eq 0 ']' 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ SHELL=/bin/bash 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorgrep.sh 2026-03-31T21:03:26.129 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'grep=grep --color=auto' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'egrep=egrep --color=auto' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'fgrep=fgrep --color=auto' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorls.sh 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' '!' -t 0 ']' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ return 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorsysstat.sh 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ export S_COLORS=auto 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ S_COLORS=auto 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorxzgrep.sh 2026-03-31T21:03:26.131 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorzgrep.sh 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2026-03-31T21:03:26.133 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zgrep=zgrep --color=auto' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zegrep=zegrep --color=auto' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/debuginfod.sh 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ prefix=/usr 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z /etc/keys/ima: ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset prefix 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/gawk.sh 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/lang.sh 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup 2026-03-31T21:03:26.135 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/locale 2026-03-31T21:03:26.136 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:03:26.136 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG_backup=en_US.UTF-8 2026-03-31T21:03:26.136 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:03:26.136 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /etc/locale.conf ']' 2026-03-31T21:03:26.136 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -x /usr/bin/sed ']' 2026-03-31T21:03:26.136 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ eval export LANG=en_US.UTF-8 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++++ export LANG=en_US.UTF-8 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++++ LANG=en_US.UTF-8 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG=en_US.UTF-8 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup config 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' dumb = linux ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/less.sh 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/which2.sh 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -r /proc/77788/exe ']' 2026-03-31T21:03:26.137 INFO:teuthology.orchestra.run.vm00.stderr:+++++ readlink /proc/77788/exe 2026-03-31T21:03:26.138 INFO:teuthology.orchestra.run.vm00.stderr:++++ basename /usr/bin/bash 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+++ SHELLNAME=bash 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+++ case "$SHELLNAME" in 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_declare='declare -f' 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_opt=-f 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+++ export which_declare 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+++ export -f which 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:++ unset i 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:++ unset -f pathmunge 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+ export PATH 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:+ unset rc 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list 2026-03-31T21:03:26.139 INFO:teuthology.orchestra.run.vm00.stderr:++ grep '"oid":"test"' 2026-03-31T21:03:26.874 INFO:teuthology.orchestra.run.vm00.stderr:+ object='["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' 2026-03-31T21:03:26.874 INFO:teuthology.orchestra.run.vm00.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 '["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' rm-attr hinfo_key 2026-03-31T21:03:28.774 INFO:tasks.ceph.osd.0:Restarting daemon 2026-03-31T21:03:28.774 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-03-31T21:03:28.816 INFO:tasks.ceph.osd.0:Started 2026-03-31T21:03:28.816 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:no valid command found; 10 closest matches: 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:0 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:1 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:2 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:abort 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:assert 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:bluestore collections 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:bluestore compression stats [] 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:bluestore list [] [] 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:bluestore onode metadata 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:config diff 2026-03-31T21:03:28.883 INFO:teuthology.orchestra.run.vm00.stderr:admin_socket: invalid command 2026-03-31T21:03:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-03-31T21:03:28.885 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-03-31T21:03:29.039 INFO:tasks.ceph.osd.0.vm00.stderr:2026-03-31T21:03:29.037+0000 7f2390413900 -1 Falling back to public interface 2026-03-31T21:03:29.466 INFO:tasks.ceph.osd.0.vm00.stderr:2026-03-31T21:03:29.463+0000 7f2390413900 -1 osd.0 34 log_to_monitors true 2026-03-31T21:03:29.531 INFO:tasks.ceph.osd.0.vm00.stderr:2026-03-31T21:03:29.529+0000 7f238695a640 -1 osd.0 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:03:33.886 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-03-31T21:03:33.963 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:03:33.963 INFO:teuthology.orchestra.run.vm00.stdout: "ops": [], 2026-03-31T21:03:33.963 INFO:teuthology.orchestra.run.vm00.stdout: "num_ops": 0 2026-03-31T21:03:33.963 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:03:33.970 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.0 to be up 2026-03-31T21:03:33.970 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:03:34.158 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:34.158 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":39,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:03:31.527400+0000","last_up_change":"2026-03-31T21:03:30.524842+0000","last_in_change":"2026-03-31T21:02:26.197000+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":34,"up_from":38,"up_thru":38,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":681364674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":681364674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":681364674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":681364674}]},"public_addr":"192.168.123.100:6812/681364674","cluster_addr":"192.168.123.100:6813/681364674","heartbeat_back_addr":"192.168.123.100:6815/681364674","heartbeat_front_addr":"192.168.123.100:6814/681364674","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":23,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":26,"up_thru":38,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":2551005687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":2551005687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":2551005687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":2551005687}]},"public_addr":"192.168.123.100:6804/2551005687","cluster_addr":"192.168.123.100:6805/2551005687","heartbeat_back_addr":"192.168.123.100:6807/2551005687","heartbeat_front_addr":"192.168.123.100:6806/2551005687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.978308+0000","dead_epoch":35},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":65536,"last_purged_snaps_scrub":"2026-03-31T21:00:59.991163+0000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.868703+0000","dead_epoch":23}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:03:34.169 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.0 is up 2026-03-31T21:03:34.170 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2026-03-31T21:03:34.369 INFO:tasks.ceph.mon.a.vm00.stderr:2026-03-31T21:03:34.367+0000 7fd706c0c640 -1 mon.a@0(leader).osd e39 definitely_dead 0 2026-03-31T21:03:34.873 INFO:teuthology.orchestra.run.vm00.stderr:marked in osd.1. 2026-03-31T21:03:34.889 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.0 to osd.1 complete 2026-03-31T21:03:39.890 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:03:39.891 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:03:39.891 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:03:39.891 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:03:40.013 INFO:teuthology.orchestra.run.vm00.stdout:163208757252 2026-03-31T21:03:40.014 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:40.018 INFO:teuthology.orchestra.run.vm00.stdout:111669149722 2026-03-31T21:03:40.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:40.020 INFO:teuthology.orchestra.run.vm00.stdout:38654705711 2026-03-31T21:03:40.020 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:40.186 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:03:40.184+0000 7fd00affd640 0 --2- 192.168.123.100:0/4288934632 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7fd00c083090 0x7fd00c082150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:03:40.216 INFO:teuthology.orchestra.run.vm00.stdout:90194313255 2026-03-31T21:03:40.217 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:40.264 INFO:teuthology.orchestra.run.vm00.stdout:163208757251 2026-03-31T21:03:40.279 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757252 got 163208757251 for osd.0 2026-03-31T21:03:40.290 INFO:teuthology.orchestra.run.vm00.stdout:111669149721 2026-03-31T21:03:40.294 INFO:teuthology.orchestra.run.vm00.stdout:38654705710 2026-03-31T21:03:40.307 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149722 got 111669149721 for osd.3 2026-03-31T21:03:40.310 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705711 got 38654705710 for osd.2 2026-03-31T21:03:40.444 INFO:teuthology.orchestra.run.vm00.stdout:90194313254 2026-03-31T21:03:40.457 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313255 got 90194313254 for osd.1 2026-03-31T21:03:41.280 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:41.308 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:41.310 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:41.459 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:41.514 INFO:teuthology.orchestra.run.vm00.stdout:163208757251 2026-03-31T21:03:41.533 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757252 got 163208757251 for osd.0 2026-03-31T21:03:41.538 INFO:teuthology.orchestra.run.vm00.stdout:111669149721 2026-03-31T21:03:41.551 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149722 got 111669149721 for osd.3 2026-03-31T21:03:41.551 INFO:teuthology.orchestra.run.vm00.stdout:38654705710 2026-03-31T21:03:41.562 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705711 got 38654705710 for osd.2 2026-03-31T21:03:41.744 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:03:41.742+0000 7f3882575640 0 --2- 192.168.123.100:0/3528664841 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f3884055a50 0x7f3884114b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:03:41.873 INFO:teuthology.orchestra.run.vm00.stdout:90194313254 2026-03-31T21:03:41.886 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313255 got 90194313254 for osd.1 2026-03-31T21:03:42.534 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:42.551 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:42.563 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:42.749 INFO:teuthology.orchestra.run.vm00.stdout:163208757252 2026-03-31T21:03:42.761 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757252 got 163208757252 for osd.0 2026-03-31T21:03:42.761 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:42.792 INFO:teuthology.orchestra.run.vm00.stdout:38654705711 2026-03-31T21:03:42.794 INFO:teuthology.orchestra.run.vm00.stdout:111669149722 2026-03-31T21:03:42.805 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705711 got 38654705711 for osd.2 2026-03-31T21:03:42.806 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:42.807 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149722 got 111669149722 for osd.3 2026-03-31T21:03:42.807 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:42.887 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:43.100 INFO:teuthology.orchestra.run.vm00.stdout:90194313256 2026-03-31T21:03:43.109 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313255 got 90194313256 for osd.1 2026-03-31T21:03:43.109 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:43.109 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:43.302 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:43.303 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:43.315 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2026-03-31T21:03:41.810194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13488,"num_bytes_recovered":110774794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":455780,"kb_used_data":215900,"kb_used_omap":68,"kb_used_meta":239739,"kb_avail":377031580,"statfs":{"total":386547056640,"available":386080337920,"internally_reserved":0,"allocated":221081600,"data_stored":220187561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69639,"internal_metadata":245493753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":1294848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.934116"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40480,"reported_epoch":42,"state":"peering","last_fresh":"2026-03-31T21:03:39.353098+0000","last_change":"2026-03-31T21:03:39.907681+0000","last_active":"2026-03-31T21:03:39.907681+0000","last_peered":"2026-03-31T21:03:39.353098+0000","last_clean":"2026-03-31T21:03:39.353098+0000","last_became_active":"2026-03-31T21:03:36.885435+0000","last_became_peered":"2026-03-31T21:03:36.885435+0000","last_unstale":"2026-03-31T21:03:39.353098+0000","last_undegraded":"2026-03-31T21:03:39.353098+0000","last_fullsized":"2026-03-31T21:03:39.353098+0000","mapping_epoch":43,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":23,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.904327+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:03:40.904327+0000","last_peered":"2026-03-31T21:03:40.904327+0000","last_clean":"2026-03-31T21:03:40.904327+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:03:40.904327+0000","last_undegraded":"2026-03-31T21:03:40.904327+0000","last_fullsized":"2026-03-31T21:03:40.904327+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218996736,"data_stored":218996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83588,"kb_used_data":53812,"kb_used_omap":15,"kb_used_meta":29744,"kb_avail":94288252,"statfs":{"total":96636764160,"available":96551170048,"internally_reserved":0,"allocated":55103488,"data_stored":54883277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16205,"internal_metadata":30458035},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98692,"kb_used_data":53812,"kb_used_omap":24,"kb_used_meta":44839,"kb_avail":94273148,"statfs":{"total":96636764160,"available":96535703552,"internally_reserved":0,"allocated":55103488,"data_stored":54883277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25063,"internal_metadata":45915673},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313256,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189844,"kb_used_data":54140,"kb_used_omap":19,"kb_used_meta":135660,"kb_avail":94181996,"statfs":{"total":96636764160,"available":96442363904,"internally_reserved":0,"allocated":55439360,"data_stored":55209538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20051,"internal_metadata":138916269},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757252,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83656,"kb_used_data":54136,"kb_used_omap":8,"kb_used_meta":29495,"kb_avail":94288184,"statfs":{"total":96636764160,"available":96551100416,"internally_reserved":0,"allocated":55435264,"data_stored":55211469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8320,"internal_metadata":30203776},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:43.315 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40480, 'reported_epoch': 42, 'state': 'peering', 'last_fresh': '2026-03-31T21:03:39.353098+0000', 'last_change': '2026-03-31T21:03:39.907681+0000', 'last_active': '2026-03-31T21:03:39.907681+0000', 'last_peered': '2026-03-31T21:03:39.353098+0000', 'last_clean': '2026-03-31T21:03:39.353098+0000', 'last_became_active': '2026-03-31T21:03:36.885435+0000', 'last_became_peered': '2026-03-31T21:03:36.885435+0000', 'last_unstale': '2026-03-31T21:03:39.353098+0000', 'last_undegraded': '2026-03-31T21:03:39.353098+0000', 'last_fullsized': '2026-03-31T21:03:39.353098+0000', 'mapping_epoch': 43, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 42, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13486, 'num_bytes_recovered': 110446602, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:43.315 DEBUG:tasks.ec_inconsistent_hinfo:not active 2026-03-31T21:03:48.316 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:03:48.317 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:03:48.317 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:03:48.317 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:03:48.430 INFO:teuthology.orchestra.run.vm00.stdout:38654705714 2026-03-31T21:03:48.430 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:48.440 INFO:teuthology.orchestra.run.vm00.stdout:111669149725 2026-03-31T21:03:48.440 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:48.440 INFO:teuthology.orchestra.run.vm00.stdout:90194313258 2026-03-31T21:03:48.440 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:48.443 INFO:teuthology.orchestra.run.vm00.stdout:163208757254 2026-03-31T21:03:48.443 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:48.651 INFO:teuthology.orchestra.run.vm00.stdout:38654705712 2026-03-31T21:03:48.674 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705714 got 38654705712 for osd.2 2026-03-31T21:03:48.714 INFO:teuthology.orchestra.run.vm00.stdout:90194313257 2026-03-31T21:03:48.715 INFO:teuthology.orchestra.run.vm00.stdout:111669149724 2026-03-31T21:03:48.728 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313258 got 90194313257 for osd.1 2026-03-31T21:03:48.732 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149725 got 111669149724 for osd.3 2026-03-31T21:03:48.764 INFO:teuthology.orchestra.run.vm00.stdout:163208757253 2026-03-31T21:03:48.778 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757254 got 163208757253 for osd.0 2026-03-31T21:03:49.675 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:49.728 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:49.762 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:49.779 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:49.903 INFO:teuthology.orchestra.run.vm00.stdout:38654705712 2026-03-31T21:03:49.921 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705714 got 38654705712 for osd.2 2026-03-31T21:03:50.002 INFO:teuthology.orchestra.run.vm00.stdout:111669149724 2026-03-31T21:03:50.015 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149725 got 111669149724 for osd.3 2026-03-31T21:03:50.041 INFO:teuthology.orchestra.run.vm00.stdout:163208757253 2026-03-31T21:03:50.068 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757254 got 163208757253 for osd.0 2026-03-31T21:03:50.452 INFO:teuthology.orchestra.run.vm00.stdout:90194313257 2026-03-31T21:03:50.466 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313258 got 90194313257 for osd.1 2026-03-31T21:03:50.921 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:51.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:51.069 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:51.137 INFO:teuthology.orchestra.run.vm00.stdout:38654705714 2026-03-31T21:03:51.159 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705714 got 38654705714 for osd.2 2026-03-31T21:03:51.159 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:51.232 INFO:teuthology.orchestra.run.vm00.stdout:111669149725 2026-03-31T21:03:51.245 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149725 got 111669149725 for osd.3 2026-03-31T21:03:51.245 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:51.284 INFO:teuthology.orchestra.run.vm00.stdout:163208757255 2026-03-31T21:03:51.296 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757254 got 163208757255 for osd.0 2026-03-31T21:03:51.296 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:51.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:51.666 INFO:teuthology.orchestra.run.vm00.stdout:90194313258 2026-03-31T21:03:51.678 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313258 got 90194313258 for osd.1 2026-03-31T21:03:51.678 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:51.678 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:51.881 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:51.881 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:51.894 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2026-03-31T21:03:51.811730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13488,"num_bytes_recovered":110774794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":402360,"kb_used_data":162480,"kb_used_omap":71,"kb_used_meta":239736,"kb_avail":377085000,"statfs":{"total":386547056640,"available":386135040000,"internally_reserved":0,"allocated":166379520,"data_stored":165456648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73549,"internal_metadata":245489843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.908706"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40490,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.909099+0000","last_change":"2026-03-31T21:03:40.909057+0000","last_active":"2026-03-31T21:03:40.909099+0000","last_peered":"2026-03-31T21:03:40.909099+0000","last_clean":"2026-03-31T21:03:40.909099+0000","last_became_active":"2026-03-31T21:03:40.908817+0000","last_became_peered":"2026-03-31T21:03:40.908817+0000","last_unstale":"2026-03-31T21:03:40.909099+0000","last_undegraded":"2026-03-31T21:03:40.909099+0000","last_fullsized":"2026-03-31T21:03:40.909099+0000","mapping_epoch":43,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T04:08:22.986879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":23,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.904327+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:03:40.904327+0000","last_peered":"2026-03-31T21:03:40.904327+0000","last_clean":"2026-03-31T21:03:40.904327+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:03:40.904327+0000","last_undegraded":"2026-03-31T21:03:40.904327+0000","last_fullsized":"2026-03-31T21:03:40.904327+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218996736,"data_stored":218996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83604,"kb_used_data":53828,"kb_used_omap":17,"kb_used_meta":29742,"kb_avail":94288236,"statfs":{"total":96636764160,"available":96551153664,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17509,"internal_metadata":30456731},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98708,"kb_used_data":53828,"kb_used_omap":25,"kb_used_meta":44838,"kb_avail":94273132,"statfs":{"total":96636764160,"available":96535687168,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26365,"internal_metadata":45914371},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189844,"kb_used_data":54140,"kb_used_omap":19,"kb_used_meta":135660,"kb_avail":94181996,"statfs":{"total":96636764160,"available":96442363904,"internally_reserved":0,"allocated":55439360,"data_stored":55209538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20051,"internal_metadata":138916269},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30204,"kb_used_data":684,"kb_used_omap":9,"kb_used_meta":29494,"kb_avail":94341636,"statfs":{"total":96636764160,"available":96605835264,"internally_reserved":0,"allocated":700416,"data_stored":467010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9624,"internal_metadata":30202472},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:51.894 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40490, 'reported_epoch': 44, 'state': 'active+clean', 'last_fresh': '2026-03-31T21:03:40.909099+0000', 'last_change': '2026-03-31T21:03:40.909057+0000', 'last_active': '2026-03-31T21:03:40.909099+0000', 'last_peered': '2026-03-31T21:03:40.909099+0000', 'last_clean': '2026-03-31T21:03:40.909099+0000', 'last_became_active': '2026-03-31T21:03:40.908817+0000', 'last_became_peered': '2026-03-31T21:03:40.908817+0000', 'last_unstale': '2026-03-31T21:03:40.909099+0000', 'last_undegraded': '2026-03-31T21:03:40.909099+0000', 'last_fullsized': '2026-03-31T21:03:40.909099+0000', 'mapping_epoch': 43, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-04-02T04:08:22.986879+0000', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13486, 'num_bytes_recovered': 110446602, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:51.894 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-03-31T21:03:51.894 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-03-31T21:03:51.894 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:52.091 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:52.091 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:52.102 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2026-03-31T21:03:51.811730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13488,"num_bytes_recovered":110774794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":402360,"kb_used_data":162480,"kb_used_omap":71,"kb_used_meta":239736,"kb_avail":377085000,"statfs":{"total":386547056640,"available":386135040000,"internally_reserved":0,"allocated":166379520,"data_stored":165456648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73549,"internal_metadata":245489843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.908706"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40490,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.909099+0000","last_change":"2026-03-31T21:03:40.909057+0000","last_active":"2026-03-31T21:03:40.909099+0000","last_peered":"2026-03-31T21:03:40.909099+0000","last_clean":"2026-03-31T21:03:40.909099+0000","last_became_active":"2026-03-31T21:03:40.908817+0000","last_became_peered":"2026-03-31T21:03:40.908817+0000","last_unstale":"2026-03-31T21:03:40.909099+0000","last_undegraded":"2026-03-31T21:03:40.909099+0000","last_fullsized":"2026-03-31T21:03:40.909099+0000","mapping_epoch":43,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T04:08:22.986879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":23,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.904327+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:03:40.904327+0000","last_peered":"2026-03-31T21:03:40.904327+0000","last_clean":"2026-03-31T21:03:40.904327+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:03:40.904327+0000","last_undegraded":"2026-03-31T21:03:40.904327+0000","last_fullsized":"2026-03-31T21:03:40.904327+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218996736,"data_stored":218996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83604,"kb_used_data":53828,"kb_used_omap":17,"kb_used_meta":29742,"kb_avail":94288236,"statfs":{"total":96636764160,"available":96551153664,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17509,"internal_metadata":30456731},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98708,"kb_used_data":53828,"kb_used_omap":25,"kb_used_meta":44838,"kb_avail":94273132,"statfs":{"total":96636764160,"available":96535687168,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26365,"internal_metadata":45914371},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189844,"kb_used_data":54140,"kb_used_omap":19,"kb_used_meta":135660,"kb_avail":94181996,"statfs":{"total":96636764160,"available":96442363904,"internally_reserved":0,"allocated":55439360,"data_stored":55209538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20051,"internal_metadata":138916269},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30204,"kb_used_data":684,"kb_used_omap":9,"kb_used_meta":29494,"kb_avail":94341636,"statfs":{"total":96636764160,"available":96605835264,"internally_reserved":0,"allocated":700416,"data_stored":467010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9624,"internal_metadata":30202472},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:52.103 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:52.295 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:52.295 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:52.308 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2026-03-31T21:03:51.811730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13488,"num_bytes_recovered":110774794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":402360,"kb_used_data":162480,"kb_used_omap":71,"kb_used_meta":239736,"kb_avail":377085000,"statfs":{"total":386547056640,"available":386135040000,"internally_reserved":0,"allocated":166379520,"data_stored":165456648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73549,"internal_metadata":245489843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.908706"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40490,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.909099+0000","last_change":"2026-03-31T21:03:40.909057+0000","last_active":"2026-03-31T21:03:40.909099+0000","last_peered":"2026-03-31T21:03:40.909099+0000","last_clean":"2026-03-31T21:03:40.909099+0000","last_became_active":"2026-03-31T21:03:40.908817+0000","last_became_peered":"2026-03-31T21:03:40.908817+0000","last_unstale":"2026-03-31T21:03:40.909099+0000","last_undegraded":"2026-03-31T21:03:40.909099+0000","last_fullsized":"2026-03-31T21:03:40.909099+0000","mapping_epoch":43,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T04:08:22.986879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":23,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.904327+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:03:40.904327+0000","last_peered":"2026-03-31T21:03:40.904327+0000","last_clean":"2026-03-31T21:03:40.904327+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:03:40.904327+0000","last_undegraded":"2026-03-31T21:03:40.904327+0000","last_fullsized":"2026-03-31T21:03:40.904327+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218996736,"data_stored":218996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83604,"kb_used_data":53828,"kb_used_omap":17,"kb_used_meta":29742,"kb_avail":94288236,"statfs":{"total":96636764160,"available":96551153664,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17509,"internal_metadata":30456731},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98708,"kb_used_data":53828,"kb_used_omap":25,"kb_used_meta":44838,"kb_avail":94273132,"statfs":{"total":96636764160,"available":96535687168,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26365,"internal_metadata":45914371},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189844,"kb_used_data":54140,"kb_used_omap":19,"kb_used_meta":135660,"kb_avail":94181996,"statfs":{"total":96636764160,"available":96442363904,"internally_reserved":0,"allocated":55439360,"data_stored":55209538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20051,"internal_metadata":138916269},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757255,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30204,"kb_used_data":684,"kb_used_omap":9,"kb_used_meta":29494,"kb_avail":94341636,"statfs":{"total":96636764160,"available":96605835264,"internally_reserved":0,"allocated":700416,"data_stored":467010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9624,"internal_metadata":30202472},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:52.308 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-03-31T21:03:52.308 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:03:52.308 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:03:52.308 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:03:52.309 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:03:52.422 INFO:teuthology.orchestra.run.vm00.stdout:90194313260 2026-03-31T21:03:52.423 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:52.423 INFO:teuthology.orchestra.run.vm00.stdout:38654705715 2026-03-31T21:03:52.423 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:52.423 INFO:teuthology.orchestra.run.vm00.stdout:163208757256 2026-03-31T21:03:52.424 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:52.432 INFO:teuthology.orchestra.run.vm00.stdout:111669149727 2026-03-31T21:03:52.432 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:52.644 INFO:teuthology.orchestra.run.vm00.stdout:111669149725 2026-03-31T21:03:52.647 INFO:teuthology.orchestra.run.vm00.stdout:90194313258 2026-03-31T21:03:52.652 INFO:teuthology.orchestra.run.vm00.stdout:38654705714 2026-03-31T21:03:52.656 INFO:teuthology.orchestra.run.vm00.stdout:163208757255 2026-03-31T21:03:52.659 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149727 got 111669149725 for osd.3 2026-03-31T21:03:52.667 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313260 got 90194313258 for osd.1 2026-03-31T21:03:52.668 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757256 got 163208757255 for osd.0 2026-03-31T21:03:52.668 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705714 for osd.2 2026-03-31T21:03:53.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:53.668 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:53.669 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:53.670 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:53.903 INFO:teuthology.orchestra.run.vm00.stdout:111669149725 2026-03-31T21:03:53.917 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149727 got 111669149725 for osd.3 2026-03-31T21:03:53.929 INFO:teuthology.orchestra.run.vm00.stdout:163208757255 2026-03-31T21:03:53.942 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757256 got 163208757255 for osd.0 2026-03-31T21:03:53.953 INFO:teuthology.orchestra.run.vm00.stdout:38654705714 2026-03-31T21:03:53.965 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705714 for osd.2 2026-03-31T21:03:53.965 INFO:teuthology.orchestra.run.vm00.stdout:90194313259 2026-03-31T21:03:53.978 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313260 got 90194313259 for osd.1 2026-03-31T21:03:54.918 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:03:54.943 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:03:54.966 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:03:54.979 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:03:55.169 INFO:teuthology.orchestra.run.vm00.stdout:111669149727 2026-03-31T21:03:55.187 INFO:teuthology.orchestra.run.vm00.stdout:163208757256 2026-03-31T21:03:55.190 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 111669149727 got 111669149727 for osd.3 2026-03-31T21:03:55.191 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:55.204 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757256 got 163208757256 for osd.0 2026-03-31T21:03:55.204 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:55.225 INFO:teuthology.orchestra.run.vm00.stdout:38654705716 2026-03-31T21:03:55.238 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705716 for osd.2 2026-03-31T21:03:55.238 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:55.257 INFO:teuthology.orchestra.run.vm00.stdout:90194313260 2026-03-31T21:03:55.269 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313260 got 90194313260 for osd.1 2026-03-31T21:03:55.270 DEBUG:teuthology.parallel:result is None 2026-03-31T21:03:55.270 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:03:55.535 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:55.535 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:03:55.550 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2026-03-31T21:03:53.812013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13488,"num_bytes_recovered":110774794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":402360,"kb_used_data":162480,"kb_used_omap":71,"kb_used_meta":239736,"kb_avail":377085000,"statfs":{"total":386547056640,"available":386135040000,"internally_reserved":0,"allocated":166379520,"data_stored":165456648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":73549,"internal_metadata":245489843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001819"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40490,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.909099+0000","last_change":"2026-03-31T21:03:40.909057+0000","last_active":"2026-03-31T21:03:40.909099+0000","last_peered":"2026-03-31T21:03:40.909099+0000","last_clean":"2026-03-31T21:03:40.909099+0000","last_became_active":"2026-03-31T21:03:40.908817+0000","last_became_peered":"2026-03-31T21:03:40.908817+0000","last_unstale":"2026-03-31T21:03:40.909099+0000","last_undegraded":"2026-03-31T21:03:40.909099+0000","last_fullsized":"2026-03-31T21:03:40.909099+0000","mapping_epoch":43,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T04:08:22.986879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":23,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-03-31T21:03:40.904327+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:03:40.904327+0000","last_peered":"2026-03-31T21:03:40.904327+0000","last_clean":"2026-03-31T21:03:40.904327+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:03:40.904327+0000","last_undegraded":"2026-03-31T21:03:40.904327+0000","last_fullsized":"2026-03-31T21:03:40.904327+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13486,"num_bytes_recovered":110446602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218996736,"data_stored":218996736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83604,"kb_used_data":53828,"kb_used_omap":17,"kb_used_meta":29742,"kb_avail":94288236,"statfs":{"total":96636764160,"available":96551153664,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17509,"internal_metadata":30456731},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98708,"kb_used_data":53828,"kb_used_omap":25,"kb_used_meta":44838,"kb_avail":94273132,"statfs":{"total":96636764160,"available":96535687168,"internally_reserved":0,"allocated":55119872,"data_stored":54890050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26365,"internal_metadata":45914371},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189844,"kb_used_data":54140,"kb_used_omap":19,"kb_used_meta":135660,"kb_avail":94181996,"statfs":{"total":96636764160,"available":96442363904,"internally_reserved":0,"allocated":55439360,"data_stored":55209538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20051,"internal_metadata":138916269},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757256,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30204,"kb_used_data":684,"kb_used_omap":9,"kb_used_meta":29494,"kb_avail":94341636,"statfs":{"total":96636764160,"available":96605835264,"internally_reserved":0,"allocated":700416,"data_stored":467010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9624,"internal_metadata":30202472},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:03:55.550 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40490, 'reported_epoch': 44, 'state': 'active+clean', 'last_fresh': '2026-03-31T21:03:40.909099+0000', 'last_change': '2026-03-31T21:03:40.909057+0000', 'last_active': '2026-03-31T21:03:40.909099+0000', 'last_peered': '2026-03-31T21:03:40.909099+0000', 'last_clean': '2026-03-31T21:03:40.909099+0000', 'last_became_active': '2026-03-31T21:03:40.908817+0000', 'last_became_peered': '2026-03-31T21:03:40.908817+0000', 'last_unstale': '2026-03-31T21:03:40.909099+0000', 'last_undegraded': '2026-03-31T21:03:40.909099+0000', 'last_fullsized': '2026-03-31T21:03:40.909099+0000', 'mapping_epoch': 43, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 44, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-04-02T04:08:22.986879+0000', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13486, 'num_bytes_recovered': 110446602, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:03:55.550 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-03-31T21:03:55.875 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:55.886 INFO:teuthology.orchestra.run.vm00.stdout:{"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":178,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1774991010,"num_in_osds":4,"osd_in_since":1774991014,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":13369,"data_bytes":109807460,"bytes_used":412016640,"bytes_avail":386135040000,"bytes_total":386547056640},"fsmap":{"epoch":1,"btime":"2026-03-31T21:00:56:876132+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":6,"modified":"2026-03-31T21:03:33.809124+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2026-03-31T21:03:55.887 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'edf0be71-aa1c-40ef-819b-fbd038c1604f', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 4, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 178, 'monmap': {'epoch': 1, 'min_mon_release_name': 'tentacle', 'num_mons': 3}, 'osdmap': {'epoch': 44, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1774991010, 'num_in_osds': 4, 'osd_in_since': 1774991014, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 13369, 'data_bytes': 109807460, 'bytes_used': 412016640, 'bytes_avail': 386135040000, 'bytes_total': 386547056640}, 'fsmap': {'epoch': 1, 'btime': '2026-03-31T21:00:56:876132+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs'], 'services': {}}, 'servicemap': {'epoch': 6, 'modified': '2026-03-31T21:03:33.809124+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2026-03-31T21:03:55.888 DEBUG:tasks.ec_inconsistent_hinfo:there are 0 unfound objects 2026-03-31T21:03:55.888 INFO:tasks.ec_inconsistent_hinfo:remove hash info from two shards and test backfill 2026-03-31T21:03:55.888 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:03:56.138 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:56.138 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":44,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:03:40.900523+0000","last_up_change":"2026-03-31T21:03:30.524842+0000","last_in_change":"2026-03-31T21:03:34.862528+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":34,"up_from":38,"up_thru":38,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":681364674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":681364674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":681364674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":681364674}]},"public_addr":"192.168.123.100:6812/681364674","cluster_addr":"192.168.123.100:6813/681364674","heartbeat_back_addr":"192.168.123.100:6815/681364674","heartbeat_front_addr":"192.168.123.100:6814/681364674","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":40,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":26,"up_thru":43,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":2551005687}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":2551005687}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":2551005687}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":2551005687}]},"public_addr":"192.168.123.100:6804/2551005687","cluster_addr":"192.168.123.100:6805/2551005687","heartbeat_back_addr":"192.168.123.100:6807/2551005687","heartbeat_front_addr":"192.168.123.100:6806/2551005687","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.978308+0000","dead_epoch":35},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.991163+0000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.868703+0000","dead_epoch":23}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:03:56.149 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-03-31T21:03:56.483 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:03:56.495 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":44,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-03-31T21:03:56.496 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-03-31T21:03:56.496 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:03:56.539 INFO:tasks.ceph.osd.3:Stopped 2026-03-31T21:03:56.539 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key 2026-03-31T21:03:56.539 DEBUG:teuthology.orchestra.run.vm00:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"'"'"oid":"test"'"'"') 2026-03-31T21:03:56.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key' 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -f /etc/bashrc ']' 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:+ . /etc/bashrc 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -z '' ']' 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ BASHRCSOURCED=Y 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ shopt -q login_shell 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:+++ umask 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' 0022 -eq 0 ']' 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ SHELL=/bin/bash 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorgrep.sh 2026-03-31T21:03:56.552 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'grep=grep --color=auto' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'egrep=egrep --color=auto' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'fgrep=fgrep --color=auto' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorls.sh 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' '!' -t 0 ']' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ return 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorsysstat.sh 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ export S_COLORS=auto 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ S_COLORS=auto 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorxzgrep.sh 2026-03-31T21:03:56.554 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorzgrep.sh 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2026-03-31T21:03:56.556 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zgrep=zgrep --color=auto' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zegrep=zegrep --color=auto' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/debuginfod.sh 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ prefix=/usr 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z /etc/keys/ima: ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset prefix 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/gawk.sh 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/lang.sh 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup 2026-03-31T21:03:56.558 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/locale 2026-03-31T21:03:56.559 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:03:56.559 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG_backup=en_US.UTF-8 2026-03-31T21:03:56.559 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:03:56.559 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /etc/locale.conf ']' 2026-03-31T21:03:56.559 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -x /usr/bin/sed ']' 2026-03-31T21:03:56.559 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ eval export LANG=en_US.UTF-8 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++++ export LANG=en_US.UTF-8 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++++ LANG=en_US.UTF-8 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG=en_US.UTF-8 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup config 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' dumb = linux ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/less.sh 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/which2.sh 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -r /proc/82534/exe ']' 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:+++++ readlink /proc/82534/exe 2026-03-31T21:03:56.560 INFO:teuthology.orchestra.run.vm00.stderr:++++ basename /usr/bin/bash 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+++ SHELLNAME=bash 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+++ case "$SHELLNAME" in 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_declare='declare -f' 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_opt=-f 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+++ export which_declare 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+++ export -f which 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:++ unset i 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:++ unset -f pathmunge 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+ export PATH 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:+ unset rc 2026-03-31T21:03:56.561 INFO:teuthology.orchestra.run.vm00.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list 2026-03-31T21:03:56.562 INFO:teuthology.orchestra.run.vm00.stderr:++ grep '"oid":"test"' 2026-03-31T21:03:56.987 INFO:teuthology.orchestra.run.vm00.stderr:+ object='["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' 2026-03-31T21:03:56.987 INFO:teuthology.orchestra.run.vm00.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 '["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' rm-attr hinfo_key 2026-03-31T21:03:58.003 INFO:tasks.ceph.osd.3:Restarting daemon 2026-03-31T21:03:58.003 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-03-31T21:03:58.045 INFO:tasks.ceph.osd.3:Started 2026-03-31T21:03:58.045 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:no valid command found; 10 closest matches: 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:0 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:1 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:2 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:abort 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:assert 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:bluestore collections 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:bluestore compression stats [] 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:bluestore list [] [] 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:bluestore onode metadata 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:config diff 2026-03-31T21:03:58.116 INFO:teuthology.orchestra.run.vm00.stderr:admin_socket: invalid command 2026-03-31T21:03:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-03-31T21:03:58.118 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-03-31T21:03:58.274 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:03:58.272+0000 7f1d55e0c900 -1 Falling back to public interface 2026-03-31T21:03:58.714 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:03:58.712+0000 7f1d55e0c900 -1 osd.3 44 log_to_monitors true 2026-03-31T21:03:59.408 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:03:59.406+0000 7f1d4cacd640 -1 osd.3 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:03:59.808 INFO:tasks.ceph.mgr.x.vm00.stderr:2026-03-31T21:03:59.806+0000 7f8b7d694640 -1 calc_pg_upmaps abort due to max <= 0 2026-03-31T21:04:03.119 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-03-31T21:04:03.193 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:04:03.193 INFO:teuthology.orchestra.run.vm00.stdout: "ops": [], 2026-03-31T21:04:03.193 INFO:teuthology.orchestra.run.vm00.stdout: "num_ops": 0 2026-03-31T21:04:03.193 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:04:03.198 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.3 to be up 2026-03-31T21:04:03.198 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:04:03.385 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:03.385 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":49,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:04:01.428245+0000","last_up_change":"2026-03-31T21:04:00.401342+0000","last_in_change":"2026-03-31T21:03:34.862528+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":34,"up_from":38,"up_thru":38,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":681364674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":681364674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":681364674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":681364674}]},"public_addr":"192.168.123.100:6812/681364674","cluster_addr":"192.168.123.100:6813/681364674","heartbeat_back_addr":"192.168.123.100:6815/681364674","heartbeat_front_addr":"192.168.123.100:6814/681364674","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":45,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":44,"up_from":48,"up_thru":48,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":3199358350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":3199358350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":3199358350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":3199358350}]},"public_addr":"192.168.123.100:6804/3199358350","cluster_addr":"192.168.123.100:6805/3199358350","heartbeat_back_addr":"192.168.123.100:6807/3199358350","heartbeat_front_addr":"192.168.123.100:6806/3199358350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.978308+0000","dead_epoch":35},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.991163+0000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.868703+0000","dead_epoch":45}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:04:03.396 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.3 is up 2026-03-31T21:04:03.397 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:04:03.585 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:03.585 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":49,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:04:01.428245+0000","last_up_change":"2026-03-31T21:04:00.401342+0000","last_in_change":"2026-03-31T21:03:34.862528+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":34,"up_from":38,"up_thru":38,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":681364674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":681364674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":681364674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":681364674}]},"public_addr":"192.168.123.100:6812/681364674","cluster_addr":"192.168.123.100:6813/681364674","heartbeat_back_addr":"192.168.123.100:6815/681364674","heartbeat_front_addr":"192.168.123.100:6814/681364674","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":45,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2670138042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2670138042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2670138042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2670138042}]},"public_addr":"192.168.123.100:6808/2670138042","cluster_addr":"192.168.123.100:6809/2670138042","heartbeat_back_addr":"192.168.123.100:6811/2670138042","heartbeat_front_addr":"192.168.123.100:6810/2670138042","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":44,"up_from":48,"up_thru":48,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":3199358350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":3199358350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":3199358350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":3199358350}]},"public_addr":"192.168.123.100:6804/3199358350","cluster_addr":"192.168.123.100:6805/3199358350","heartbeat_back_addr":"192.168.123.100:6807/3199358350","heartbeat_front_addr":"192.168.123.100:6806/3199358350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.978308+0000","dead_epoch":35},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.991163+0000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.868703+0000","dead_epoch":45}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:04:03.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-03-31T21:04:03.790 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:03.801 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":49,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-03-31T21:04:03.802 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-03-31T21:04:03.802 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:04:03.842 INFO:tasks.ceph.osd.2:Stopped 2026-03-31T21:04:03.842 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 "$object" rm-attr hinfo_key 2026-03-31T21:04:03.842 DEBUG:teuthology.orchestra.run.vm00:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list |grep '"'"'"oid":"test"'"'"') 2026-03-31T21:04:03.842 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 "$object" rm-attr hinfo_key' 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -f /etc/bashrc ']' 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:+ . /etc/bashrc 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -z '' ']' 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ BASHRCSOURCED=Y 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ shopt -q login_shell 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:+++ umask 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' 0022 -eq 0 ']' 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ SHELL=/bin/bash 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorgrep.sh 2026-03-31T21:04:03.856 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'grep=grep --color=auto' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'egrep=egrep --color=auto' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'fgrep=fgrep --color=auto' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorls.sh 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' '!' -t 0 ']' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ return 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorsysstat.sh 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ export S_COLORS=auto 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ S_COLORS=auto 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorxzgrep.sh 2026-03-31T21:04:03.858 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/colorzgrep.sh 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2026-03-31T21:04:03.860 INFO:teuthology.orchestra.run.vm00.stderr:+++ /usr/libexec/grepconf.sh -c 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zgrep=zgrep --color=auto' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ alias 'zegrep=zegrep --color=auto' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/debuginfod.sh 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ prefix=/usr 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z /etc/keys/ima: ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset prefix 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/gawk.sh 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/lang.sh 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup 2026-03-31T21:04:03.862 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/locale 2026-03-31T21:04:03.863 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:04:03.863 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG_backup=en_US.UTF-8 2026-03-31T21:04:03.863 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:04:03.863 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /etc/locale.conf ']' 2026-03-31T21:04:03.863 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -x /usr/bin/sed ']' 2026-03-31T21:04:03.863 INFO:teuthology.orchestra.run.vm00.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ eval export LANG=en_US.UTF-8 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++++ export LANG=en_US.UTF-8 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++++ LANG=en_US.UTF-8 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ LANG=en_US.UTF-8 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ unset LANG_backup config 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n '' ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -n en_US.UTF-8 ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' dumb = linux ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/less.sh 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ for i in /etc/profile.d/*.sh 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ '[' '' ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:++ . /etc/profile.d/which2.sh 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++ '[' -r /proc/84585/exe ']' 2026-03-31T21:04:03.864 INFO:teuthology.orchestra.run.vm00.stderr:+++++ readlink /proc/84585/exe 2026-03-31T21:04:03.865 INFO:teuthology.orchestra.run.vm00.stderr:++++ basename /usr/bin/bash 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+++ SHELLNAME=bash 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+++ case "$SHELLNAME" in 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_declare='declare -f' 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+++ which_opt=-f 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+++ export which_declare 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+++ export -f which 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:++ unset i 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:++ unset -f pathmunge 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+ export PATH 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:+ unset rc 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list 2026-03-31T21:04:03.866 INFO:teuthology.orchestra.run.vm00.stderr:++ grep '"oid":"test"' 2026-03-31T21:04:04.306 INFO:teuthology.orchestra.run.vm00.stderr:+ object='["2.0s2",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":2,"max":0}]' 2026-03-31T21:04:04.306 INFO:teuthology.orchestra.run.vm00.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 '["2.0s2",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":2,"max":0}]' rm-attr hinfo_key 2026-03-31T21:04:05.467 INFO:tasks.ceph.osd.2:Restarting daemon 2026-03-31T21:04:05.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-03-31T21:04:05.510 INFO:tasks.ceph.osd.2:Started 2026-03-31T21:04:05.510 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:no valid command found; 10 closest matches: 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:0 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:1 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:2 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:abort 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:assert 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:bluestore collections 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:bluestore compression stats [] 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:bluestore list [] [] 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:bluestore onode metadata 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:config diff 2026-03-31T21:04:05.582 INFO:teuthology.orchestra.run.vm00.stderr:admin_socket: invalid command 2026-03-31T21:04:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-03-31T21:04:05.585 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-03-31T21:04:05.711 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:04:05.709+0000 7f2744d4d900 -1 Falling back to public interface 2026-03-31T21:04:06.109 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:04:06.107+0000 7f2744d4d900 -1 osd.2 49 log_to_monitors true 2026-03-31T21:04:06.675 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:04:06.673+0000 7f273ba9a640 -1 osd.2 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-03-31T21:04:10.585 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-03-31T21:04:10.656 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:04:10.656 INFO:teuthology.orchestra.run.vm00.stdout: "ops": [], 2026-03-31T21:04:10.656 INFO:teuthology.orchestra.run.vm00.stdout: "num_ops": 0 2026-03-31T21:04:10.656 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:04:10.663 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.2 to be up 2026-03-31T21:04:10.663 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:04:10.849 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:10.849 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":54,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:04:08.961698+0000","last_up_change":"2026-03-31T21:04:07.667010+0000","last_in_change":"2026-03-31T21:03:34.862528+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":34,"up_from":38,"up_thru":38,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":681364674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":681364674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":681364674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":681364674}]},"public_addr":"192.168.123.100:6812/681364674","cluster_addr":"192.168.123.100:6813/681364674","heartbeat_back_addr":"192.168.123.100:6815/681364674","heartbeat_front_addr":"192.168.123.100:6814/681364674","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":45,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":35,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2560261943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2560261943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2560261943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2560261943}]},"public_addr":"192.168.123.100:6808/2560261943","cluster_addr":"192.168.123.100:6809/2560261943","heartbeat_back_addr":"192.168.123.100:6811/2560261943","heartbeat_front_addr":"192.168.123.100:6810/2560261943","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":44,"up_from":48,"up_thru":53,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":3199358350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":3199358350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":3199358350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":3199358350}]},"public_addr":"192.168.123.100:6804/3199358350","cluster_addr":"192.168.123.100:6805/3199358350","heartbeat_back_addr":"192.168.123.100:6807/3199358350","heartbeat_front_addr":"192.168.123.100:6806/3199358350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.978308+0000","dead_epoch":35},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.991163+0000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":50},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.868703+0000","dead_epoch":45}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:04:10.861 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.2 is up 2026-03-31T21:04:10.861 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2026-03-31T21:04:11.056 INFO:tasks.ceph.mon.a.vm00.stderr:2026-03-31T21:04:11.054+0000 7fd706c0c640 -1 mon.a@0(leader).osd e54 definitely_dead 0 2026-03-31T21:04:11.986 INFO:teuthology.orchestra.run.vm00.stderr:marked out osd.2. 2026-03-31T21:04:12.002 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.2 to osd.0 complete 2026-03-31T21:04:17.003 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:04:17.003 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:04:17.003 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:04:17.003 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:04:17.113 INFO:teuthology.orchestra.run.vm00.stdout:90194313266 2026-03-31T21:04:17.114 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:17.114 INFO:teuthology.orchestra.run.vm00.stdout:227633266692 2026-03-31T21:04:17.114 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:17.114 INFO:teuthology.orchestra.run.vm00.stdout:206158430213 2026-03-31T21:04:17.114 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:17.114 INFO:teuthology.orchestra.run.vm00.stdout:163208757262 2026-03-31T21:04:17.115 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:17.370 INFO:teuthology.orchestra.run.vm00.stdout:90194313264 2026-03-31T21:04:17.371 INFO:teuthology.orchestra.run.vm00.stdout:0 2026-03-31T21:04:17.371 INFO:teuthology.orchestra.run.vm00.stdout:206158430212 2026-03-31T21:04:17.375 INFO:teuthology.orchestra.run.vm00.stdout:163208757261 2026-03-31T21:04:17.384 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266692 got 0 for osd.2 2026-03-31T21:04:17.384 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430213 got 206158430212 for osd.3 2026-03-31T21:04:17.387 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757262 got 163208757261 for osd.0 2026-03-31T21:04:17.389 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313266 got 90194313264 for osd.1 2026-03-31T21:04:18.385 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:18.385 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:18.387 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:18.389 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:18.601 INFO:teuthology.orchestra.run.vm00.stdout:206158430213 2026-03-31T21:04:18.618 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430213 got 206158430213 for osd.3 2026-03-31T21:04:18.618 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:18.633 INFO:teuthology.orchestra.run.vm00.stdout:227633266692 2026-03-31T21:04:18.648 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266692 got 227633266692 for osd.2 2026-03-31T21:04:18.648 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:18.651 INFO:teuthology.orchestra.run.vm00.stdout:163208757262 2026-03-31T21:04:18.665 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757262 got 163208757262 for osd.0 2026-03-31T21:04:18.665 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:18.666 INFO:teuthology.orchestra.run.vm00.stdout:90194313266 2026-03-31T21:04:18.679 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313266 got 90194313266 for osd.1 2026-03-31T21:04:18.679 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:18.679 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:04:18.906 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:18.906 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:04:18.917 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2026-03-31T21:04:17.815059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13340,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13516,"num_bytes_recovered":111004170,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":304588,"kb_used_data":109076,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282810932,"statfs":{"total":289910292480,"available":289598394368,"internally_reserved":0,"allocated":111693824,"data_stored":110819970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13340,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":229376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000764"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40592,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:04:17.007643+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:04:17.007643+0000","last_peered":"2026-03-31T21:04:17.007643+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:04:17.007643+0000","last_undegraded":"2026-03-31T21:04:17.007643+0000","last_fullsized":"2026-03-31T21:04:17.007643+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13340,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13514,"num_bytes_recovered":110675978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13340,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13514,"num_bytes_recovered":110675978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":164360192,"data_stored":164360192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30420,"kb_used_data":900,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94341420,"statfs":{"total":96636764160,"available":96605614080,"internally_reserved":0,"allocated":921600,"data_stored":627926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:04:18.917 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40592, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:04:17.007643+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:04:17.007643+0000', 'last_peered': '2026-03-31T21:04:17.007643+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:04:17.007643+0000', 'last_undegraded': '2026-03-31T21:04:17.007643+0000', 'last_fullsized': '2026-03-31T21:04:17.007643+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 13340, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13514, 'num_bytes_recovered': 110675978, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:04:18.917 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:04:23.919 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:04:23.919 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:04:23.919 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:04:23.919 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:04:24.030 INFO:teuthology.orchestra.run.vm00.stdout:163208757264 2026-03-31T21:04:24.030 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:24.034 INFO:teuthology.orchestra.run.vm00.stdout:227633266694 2026-03-31T21:04:24.035 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:24.035 INFO:teuthology.orchestra.run.vm00.stdout:90194313268 2026-03-31T21:04:24.035 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:24.044 INFO:teuthology.orchestra.run.vm00.stdout:206158430215 2026-03-31T21:04:24.044 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:24.252 INFO:teuthology.orchestra.run.vm00.stdout:163208757263 2026-03-31T21:04:24.257 INFO:teuthology.orchestra.run.vm00.stdout:227633266693 2026-03-31T21:04:24.270 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757264 got 163208757263 for osd.0 2026-03-31T21:04:24.272 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266694 got 227633266693 for osd.2 2026-03-31T21:04:24.297 INFO:teuthology.orchestra.run.vm00.stdout:90194313267 2026-03-31T21:04:24.300 INFO:teuthology.orchestra.run.vm00.stdout:206158430214 2026-03-31T21:04:24.311 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313268 got 90194313267 for osd.1 2026-03-31T21:04:24.314 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430215 got 206158430214 for osd.3 2026-03-31T21:04:25.271 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:25.273 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:25.312 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:25.314 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:25.498 INFO:teuthology.orchestra.run.vm00.stdout:163208757263 2026-03-31T21:04:25.504 INFO:teuthology.orchestra.run.vm00.stdout:227633266693 2026-03-31T21:04:25.512 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757264 got 163208757263 for osd.0 2026-03-31T21:04:25.519 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266694 got 227633266693 for osd.2 2026-03-31T21:04:25.527 INFO:teuthology.orchestra.run.vm00.stdout:206158430214 2026-03-31T21:04:25.539 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430215 got 206158430214 for osd.3 2026-03-31T21:04:25.555 INFO:teuthology.orchestra.run.vm00.stdout:90194313267 2026-03-31T21:04:25.568 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313268 got 90194313267 for osd.1 2026-03-31T21:04:26.513 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:26.519 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:26.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:26.569 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:26.721 INFO:teuthology.orchestra.run.vm00.stdout:163208757265 2026-03-31T21:04:26.733 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757264 got 163208757265 for osd.0 2026-03-31T21:04:26.733 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:26.764 INFO:teuthology.orchestra.run.vm00.stdout:227633266694 2026-03-31T21:04:26.778 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266694 got 227633266694 for osd.2 2026-03-31T21:04:26.778 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:26.778 INFO:teuthology.orchestra.run.vm00.stdout:206158430216 2026-03-31T21:04:26.790 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430215 got 206158430216 for osd.3 2026-03-31T21:04:26.790 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:26.837 INFO:teuthology.orchestra.run.vm00.stdout:90194313268 2026-03-31T21:04:26.850 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313268 got 90194313268 for osd.1 2026-03-31T21:04:26.850 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:26.850 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:04:27.121 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:04:27.120+0000 7f0cb5c96640 0 --2- 192.168.123.100:0/121302073 >> v2:192.168.123.100:3300/0 conn(0x7f0cb0080310 0x7f0cb0172fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:04:27.246 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:27.246 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:04:27.258 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2026-03-31T21:04:25.816170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13268,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13588,"num_bytes_recovered":111593994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":304860,"kb_used_data":109348,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282810660,"statfs":{"total":289910292480,"available":289598115840,"internally_reserved":0,"allocated":111972352,"data_stored":111098498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13268,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.820624"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40736,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:04:24.341869+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:04:24.341869+0000","last_peered":"2026-03-31T21:04:24.341869+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:04:24.341869+0000","last_undegraded":"2026-03-31T21:04:24.341869+0000","last_fullsized":"2026-03-31T21:04:24.341869+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13268,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13586,"num_bytes_recovered":111265802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13268,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13586,"num_bytes_recovered":111265802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":164659200,"data_stored":164659200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430216,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30692,"kb_used_data":1172,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94341148,"statfs":{"total":96636764160,"available":96605335552,"internally_reserved":0,"allocated":1200128,"data_stored":906454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":413696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:04:27.258 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40736, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:04:24.341869+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:04:24.341869+0000', 'last_peered': '2026-03-31T21:04:24.341869+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:04:24.341869+0000', 'last_undegraded': '2026-03-31T21:04:24.341869+0000', 'last_fullsized': '2026-03-31T21:04:24.341869+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 13268, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13586, 'num_bytes_recovered': 111265802, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:04:27.258 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:04:32.259 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:04:32.260 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:04:32.260 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:04:32.260 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:04:32.367 INFO:teuthology.orchestra.run.vm00.stdout:163208757267 2026-03-31T21:04:32.367 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:32.369 INFO:teuthology.orchestra.run.vm00.stdout:90194313271 2026-03-31T21:04:32.369 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:32.372 INFO:teuthology.orchestra.run.vm00.stdout:227633266697 2026-03-31T21:04:32.373 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:32.379 INFO:teuthology.orchestra.run.vm00.stdout:206158430218 2026-03-31T21:04:32.379 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:32.582 INFO:teuthology.orchestra.run.vm00.stdout:163208757266 2026-03-31T21:04:32.586 INFO:teuthology.orchestra.run.vm00.stdout:90194313270 2026-03-31T21:04:32.587 INFO:teuthology.orchestra.run.vm00.stdout:206158430217 2026-03-31T21:04:32.589 INFO:teuthology.orchestra.run.vm00.stdout:227633266696 2026-03-31T21:04:32.598 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313271 got 90194313270 for osd.1 2026-03-31T21:04:32.600 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757267 got 163208757266 for osd.0 2026-03-31T21:04:32.603 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266697 got 227633266696 for osd.2 2026-03-31T21:04:32.605 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430218 got 206158430217 for osd.3 2026-03-31T21:04:33.598 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:33.600 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:33.604 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:33.606 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:33.808 INFO:teuthology.orchestra.run.vm00.stdout:163208757266 2026-03-31T21:04:33.810 INFO:teuthology.orchestra.run.vm00.stdout:90194313270 2026-03-31T21:04:33.821 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757267 got 163208757266 for osd.0 2026-03-31T21:04:33.831 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313271 got 90194313270 for osd.1 2026-03-31T21:04:33.851 INFO:teuthology.orchestra.run.vm00.stdout:206158430217 2026-03-31T21:04:33.860 INFO:teuthology.orchestra.run.vm00.stdout:227633266696 2026-03-31T21:04:33.864 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430218 got 206158430217 for osd.3 2026-03-31T21:04:33.872 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266697 got 227633266696 for osd.2 2026-03-31T21:04:34.822 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:34.832 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:34.864 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:34.873 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:35.047 INFO:teuthology.orchestra.run.vm00.stdout:163208757267 2026-03-31T21:04:35.063 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757267 got 163208757267 for osd.0 2026-03-31T21:04:35.064 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:35.082 INFO:teuthology.orchestra.run.vm00.stdout:206158430218 2026-03-31T21:04:35.085 INFO:teuthology.orchestra.run.vm00.stdout:90194313271 2026-03-31T21:04:35.097 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430218 got 206158430218 for osd.3 2026-03-31T21:04:35.098 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:35.101 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313271 got 90194313271 for osd.1 2026-03-31T21:04:35.101 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:35.134 INFO:teuthology.orchestra.run.vm00.stdout:227633266697 2026-03-31T21:04:35.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266697 got 227633266697 for osd.2 2026-03-31T21:04:35.146 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:35.146 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:04:35.382 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:35.382 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:04:35.393 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2026-03-31T21:04:33.817177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13190,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13666,"num_bytes_recovered":112232970,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":305176,"kb_used_data":109664,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282810344,"statfs":{"total":289910292480,"available":289597792256,"internally_reserved":0,"allocated":112295936,"data_stored":111422082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001524"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":40892,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:04:32.293855+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:04:32.293855+0000","last_peered":"2026-03-31T21:04:32.293855+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:04:32.293855+0000","last_undegraded":"2026-03-31T21:04:32.293855+0000","last_fullsized":"2026-03-31T21:04:32.293855+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13190,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13664,"num_bytes_recovered":111904778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13190,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13664,"num_bytes_recovered":111904778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":164974592,"data_stored":164974592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430218,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313271,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31008,"kb_used_data":1488,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94340832,"statfs":{"total":96636764160,"available":96605011968,"internally_reserved":0,"allocated":1523712,"data_stored":1230038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":729088,"data_stored":729088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:04:35.393 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 40892, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:04:32.293855+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:04:32.293855+0000', 'last_peered': '2026-03-31T21:04:32.293855+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:04:32.293855+0000', 'last_undegraded': '2026-03-31T21:04:32.293855+0000', 'last_fullsized': '2026-03-31T21:04:32.293855+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 13190, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13664, 'num_bytes_recovered': 111904778, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:04:35.393 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:04:40.394 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:04:40.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:04:40.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:04:40.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:04:40.491 INFO:teuthology.orchestra.run.vm00.stdout:163208757270 2026-03-31T21:04:40.491 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:40.504 INFO:teuthology.orchestra.run.vm00.stdout:90194313273 2026-03-31T21:04:40.504 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:40.507 INFO:teuthology.orchestra.run.vm00.stdout:206158430221 2026-03-31T21:04:40.507 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:40.513 INFO:teuthology.orchestra.run.vm00.stdout:227633266699 2026-03-31T21:04:40.513 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:40.718 INFO:teuthology.orchestra.run.vm00.stdout:163208757269 2026-03-31T21:04:40.727 INFO:teuthology.orchestra.run.vm00.stdout:90194313272 2026-03-31T21:04:40.727 INFO:teuthology.orchestra.run.vm00.stdout:206158430220 2026-03-31T21:04:40.734 INFO:teuthology.orchestra.run.vm00.stdout:227633266698 2026-03-31T21:04:40.739 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757270 got 163208757269 for osd.0 2026-03-31T21:04:40.741 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430221 got 206158430220 for osd.3 2026-03-31T21:04:40.743 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313273 got 90194313272 for osd.1 2026-03-31T21:04:40.747 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266699 got 227633266698 for osd.2 2026-03-31T21:04:41.740 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:41.741 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:41.743 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:41.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:41.948 INFO:teuthology.orchestra.run.vm00.stdout:206158430220 2026-03-31T21:04:41.951 INFO:teuthology.orchestra.run.vm00.stdout:163208757269 2026-03-31T21:04:41.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430221 got 206158430220 for osd.3 2026-03-31T21:04:41.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757270 got 163208757269 for osd.0 2026-03-31T21:04:41.995 INFO:teuthology.orchestra.run.vm00.stdout:90194313272 2026-03-31T21:04:42.006 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313273 got 90194313272 for osd.1 2026-03-31T21:04:42.067 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:04:42.065+0000 7f1b56ffd640 0 --2- 192.168.123.100:0/2926471526 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f1b5812e990 0x7f1b5812ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:04:42.191 INFO:teuthology.orchestra.run.vm00.stdout:227633266700 2026-03-31T21:04:42.201 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266699 got 227633266700 for osd.2 2026-03-31T21:04:42.201 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:42.965 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:42.965 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:43.007 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:43.161 INFO:teuthology.orchestra.run.vm00.stdout:206158430221 2026-03-31T21:04:43.174 INFO:teuthology.orchestra.run.vm00.stdout:163208757270 2026-03-31T21:04:43.179 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430221 got 206158430221 for osd.3 2026-03-31T21:04:43.179 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:43.187 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757270 got 163208757270 for osd.0 2026-03-31T21:04:43.187 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:43.225 INFO:teuthology.orchestra.run.vm00.stdout:90194313274 2026-03-31T21:04:43.236 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313273 got 90194313274 for osd.1 2026-03-31T21:04:43.236 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:43.236 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:04:43.460 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:43.460 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:04:43.469 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2026-03-31T21:04:41.818456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13111,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13745,"num_bytes_recovered":112880138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":305488,"kb_used_data":109976,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282810032,"statfs":{"total":289910292480,"available":289597472768,"internally_reserved":0,"allocated":112615424,"data_stored":111741570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001751"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":41050,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:04:40.395015+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:04:40.395015+0000","last_peered":"2026-03-31T21:04:40.395015+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:04:40.395015+0000","last_undegraded":"2026-03-31T21:04:40.395015+0000","last_fullsized":"2026-03-31T21:04:40.395015+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13111,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13743,"num_bytes_recovered":112551946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13111,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13743,"num_bytes_recovered":112551946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":165298176,"data_stored":165298176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430221,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":1800,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":1843200,"data_stored":1549526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1052672,"data_stored":1052672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:04:43.469 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 41050, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:04:40.395015+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:04:40.395015+0000', 'last_peered': '2026-03-31T21:04:40.395015+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:04:40.395015+0000', 'last_undegraded': '2026-03-31T21:04:40.395015+0000', 'last_fullsized': '2026-03-31T21:04:40.395015+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 13111, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13743, 'num_bytes_recovered': 112551946, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:04:43.470 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:04:48.470 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:04:48.471 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:04:48.471 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:04:48.471 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:04:48.571 INFO:teuthology.orchestra.run.vm00.stdout:163208757272 2026-03-31T21:04:48.571 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:48.572 INFO:teuthology.orchestra.run.vm00.stdout:90194313276 2026-03-31T21:04:48.572 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:48.575 INFO:teuthology.orchestra.run.vm00.stdout:227633266702 2026-03-31T21:04:48.575 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:48.580 INFO:teuthology.orchestra.run.vm00.stdout:206158430223 2026-03-31T21:04:48.581 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:48.782 INFO:teuthology.orchestra.run.vm00.stdout:227633266701 2026-03-31T21:04:48.786 INFO:teuthology.orchestra.run.vm00.stdout:206158430222 2026-03-31T21:04:48.789 INFO:teuthology.orchestra.run.vm00.stdout:163208757271 2026-03-31T21:04:48.799 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266702 got 227633266701 for osd.2 2026-03-31T21:04:48.799 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430223 got 206158430222 for osd.3 2026-03-31T21:04:48.800 INFO:teuthology.orchestra.run.vm00.stdout:90194313275 2026-03-31T21:04:48.808 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757272 got 163208757271 for osd.0 2026-03-31T21:04:48.814 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313276 got 90194313275 for osd.1 2026-03-31T21:04:49.800 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:49.801 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:49.808 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:49.815 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:50.011 INFO:teuthology.orchestra.run.vm00.stdout:227633266701 2026-03-31T21:04:50.025 INFO:teuthology.orchestra.run.vm00.stdout:206158430222 2026-03-31T21:04:50.028 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266702 got 227633266701 for osd.2 2026-03-31T21:04:50.039 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430223 got 206158430222 for osd.3 2026-03-31T21:04:50.060 INFO:teuthology.orchestra.run.vm00.stdout:163208757271 2026-03-31T21:04:50.065 INFO:teuthology.orchestra.run.vm00.stdout:90194313275 2026-03-31T21:04:50.073 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757272 got 163208757271 for osd.0 2026-03-31T21:04:50.081 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313276 got 90194313275 for osd.1 2026-03-31T21:04:51.029 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:51.041 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:51.074 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:51.081 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:51.243 INFO:teuthology.orchestra.run.vm00.stdout:227633266702 2026-03-31T21:04:51.263 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266702 got 227633266702 for osd.2 2026-03-31T21:04:51.263 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:51.297 INFO:teuthology.orchestra.run.vm00.stdout:206158430224 2026-03-31T21:04:51.309 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430223 got 206158430224 for osd.3 2026-03-31T21:04:51.309 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:51.326 INFO:teuthology.orchestra.run.vm00.stdout:90194313276 2026-03-31T21:04:51.337 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313276 got 90194313276 for osd.1 2026-03-31T21:04:51.337 INFO:teuthology.orchestra.run.vm00.stdout:163208757273 2026-03-31T21:04:51.337 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:51.347 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757272 got 163208757273 for osd.0 2026-03-31T21:04:51.347 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:51.347 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:04:51.566 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:51.566 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:04:51.575 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2026-03-31T21:04:49.819427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13023,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13833,"num_bytes_recovered":113601034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":305840,"kb_used_data":110328,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282809680,"statfs":{"total":289910292480,"available":289597112320,"internally_reserved":0,"allocated":112975872,"data_stored":112102018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001774"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":41226,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:04:49.353017+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:04:49.353017+0000","last_peered":"2026-03-31T21:04:49.353017+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:04:49.353017+0000","last_undegraded":"2026-03-31T21:04:49.353017+0000","last_fullsized":"2026-03-31T21:04:49.353017+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13023,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13831,"num_bytes_recovered":113272842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13023,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13831,"num_bytes_recovered":113272842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":165662720,"data_stored":165662720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430224,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313276,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757273,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":2152,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":2203648,"data_stored":1909974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1417216,"data_stored":1417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:04:51.575 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 41226, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:04:49.353017+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:04:49.353017+0000', 'last_peered': '2026-03-31T21:04:49.353017+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:04:49.353017+0000', 'last_undegraded': '2026-03-31T21:04:49.353017+0000', 'last_fullsized': '2026-03-31T21:04:49.353017+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 13023, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13831, 'num_bytes_recovered': 113272842, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:04:51.575 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:04:56.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:04:56.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:04:56.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:04:56.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:04:56.683 INFO:teuthology.orchestra.run.vm00.stdout:206158430226 2026-03-31T21:04:56.683 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:56.683 INFO:teuthology.orchestra.run.vm00.stdout:227633266704 2026-03-31T21:04:56.683 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:56.686 INFO:teuthology.orchestra.run.vm00.stdout:90194313278 2026-03-31T21:04:56.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:56.696 INFO:teuthology.orchestra.run.vm00.stdout:163208757275 2026-03-31T21:04:56.696 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:56.893 INFO:teuthology.orchestra.run.vm00.stdout:227633266703 2026-03-31T21:04:56.893 INFO:teuthology.orchestra.run.vm00.stdout:206158430225 2026-03-31T21:04:56.906 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266704 got 227633266703 for osd.2 2026-03-31T21:04:56.911 INFO:teuthology.orchestra.run.vm00.stdout:90194313277 2026-03-31T21:04:56.913 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430226 got 206158430225 for osd.3 2026-03-31T21:04:56.923 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313278 got 90194313277 for osd.1 2026-03-31T21:04:56.955 INFO:teuthology.orchestra.run.vm00.stdout:163208757274 2026-03-31T21:04:56.966 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757275 got 163208757274 for osd.0 2026-03-31T21:04:57.907 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:57.913 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:57.924 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:57.966 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:04:58.142 INFO:teuthology.orchestra.run.vm00.stdout:227633266703 2026-03-31T21:04:58.159 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266704 got 227633266703 for osd.2 2026-03-31T21:04:58.160 INFO:teuthology.orchestra.run.vm00.stdout:90194313277 2026-03-31T21:04:58.167 INFO:teuthology.orchestra.run.vm00.stdout:206158430225 2026-03-31T21:04:58.171 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313278 got 90194313277 for osd.1 2026-03-31T21:04:58.184 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430226 got 206158430225 for osd.3 2026-03-31T21:04:58.219 INFO:teuthology.orchestra.run.vm00.stdout:163208757275 2026-03-31T21:04:58.232 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757275 got 163208757275 for osd.0 2026-03-31T21:04:58.232 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:59.160 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:04:59.171 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:04:59.185 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:04:59.365 INFO:teuthology.orchestra.run.vm00.stdout:227633266705 2026-03-31T21:04:59.377 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266704 got 227633266705 for osd.2 2026-03-31T21:04:59.377 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:59.408 INFO:teuthology.orchestra.run.vm00.stdout:206158430226 2026-03-31T21:04:59.416 INFO:teuthology.orchestra.run.vm00.stdout:90194313279 2026-03-31T21:04:59.422 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430226 got 206158430226 for osd.3 2026-03-31T21:04:59.422 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:59.427 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313278 got 90194313279 for osd.1 2026-03-31T21:04:59.427 DEBUG:teuthology.parallel:result is None 2026-03-31T21:04:59.427 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:04:59.653 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:04:59.653 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:04:59.664 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2026-03-31T21:04:57.820455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12952,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13904,"num_bytes_recovered":114182666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":306112,"kb_used_data":110600,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282809408,"statfs":{"total":289910292480,"available":289596833792,"internally_reserved":0,"allocated":113254400,"data_stored":112380546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-120,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":41368,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:04:56.585872+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:04:56.585872+0000","last_peered":"2026-03-31T21:04:56.585872+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:04:56.585872+0000","last_undegraded":"2026-03-31T21:04:56.585872+0000","last_fullsized":"2026-03-31T21:04:56.585872+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12952,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13902,"num_bytes_recovered":113854474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12952,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13902,"num_bytes_recovered":113854474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":165949440,"data_stored":165949440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430226,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757275,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31944,"kb_used_data":2424,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94339896,"statfs":{"total":96636764160,"available":96604053504,"internally_reserved":0,"allocated":2482176,"data_stored":2188502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1703936,"data_stored":1703936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:04:59.664 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 41368, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:04:56.585872+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:04:56.585872+0000', 'last_peered': '2026-03-31T21:04:56.585872+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:04:56.585872+0000', 'last_undegraded': '2026-03-31T21:04:56.585872+0000', 'last_fullsized': '2026-03-31T21:04:56.585872+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12952, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13902, 'num_bytes_recovered': 113854474, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:04:59.664 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:04.665 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:04.666 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:04.666 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:04.666 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:04.770 INFO:teuthology.orchestra.run.vm00.stdout:163208757278 2026-03-31T21:05:04.771 INFO:teuthology.orchestra.run.vm00.stdout:206158430229 2026-03-31T21:05:04.771 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:04.771 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:04.771 INFO:teuthology.orchestra.run.vm00.stdout:227633266707 2026-03-31T21:05:04.771 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:04.771 INFO:teuthology.orchestra.run.vm00.stdout:90194313281 2026-03-31T21:05:04.772 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:05.021 INFO:teuthology.orchestra.run.vm00.stdout:163208757276 2026-03-31T21:05:05.023 INFO:teuthology.orchestra.run.vm00.stdout:206158430227 2026-03-31T21:05:05.023 INFO:teuthology.orchestra.run.vm00.stdout:227633266706 2026-03-31T21:05:05.025 INFO:teuthology.orchestra.run.vm00.stdout:90194313280 2026-03-31T21:05:05.039 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430229 got 206158430227 for osd.3 2026-03-31T21:05:05.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313281 got 90194313280 for osd.1 2026-03-31T21:05:05.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757278 got 163208757276 for osd.0 2026-03-31T21:05:05.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266707 got 227633266706 for osd.2 2026-03-31T21:05:06.040 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:06.042 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:06.043 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:06.043 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:06.264 INFO:teuthology.orchestra.run.vm00.stdout:206158430229 2026-03-31T21:05:06.282 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430229 got 206158430229 for osd.3 2026-03-31T21:05:06.282 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:06.291 INFO:teuthology.orchestra.run.vm00.stdout:227633266707 2026-03-31T21:05:06.302 INFO:teuthology.orchestra.run.vm00.stdout:90194313281 2026-03-31T21:05:06.304 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266707 got 227633266707 for osd.2 2026-03-31T21:05:06.305 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:06.306 INFO:teuthology.orchestra.run.vm00.stdout:163208757278 2026-03-31T21:05:06.316 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313281 got 90194313281 for osd.1 2026-03-31T21:05:06.316 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:06.321 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757278 got 163208757278 for osd.0 2026-03-31T21:05:06.321 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:06.321 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:06.564 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:06.564 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:06.575 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2026-03-31T21:05:05.821683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12872,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13984,"num_bytes_recovered":114838026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":306428,"kb_used_data":110916,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282809092,"statfs":{"total":289910292480,"available":289596510208,"internally_reserved":0,"allocated":113577984,"data_stored":112704130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-151,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001739"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":41528,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:04.745620+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:04.745620+0000","last_peered":"2026-03-31T21:05:04.745620+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:04.745620+0000","last_undegraded":"2026-03-31T21:05:04.745620+0000","last_fullsized":"2026-03-31T21:05:04.745620+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12872,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13982,"num_bytes_recovered":114509834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12872,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13982,"num_bytes_recovered":114509834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166277120,"data_stored":166277120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430229,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313281,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":2740,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":2805760,"data_stored":2512086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2031616,"data_stored":2031616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:06.575 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 41528, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:04.745620+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:04.745620+0000', 'last_peered': '2026-03-31T21:05:04.745620+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:04.745620+0000', 'last_undegraded': '2026-03-31T21:05:04.745620+0000', 'last_fullsized': '2026-03-31T21:05:04.745620+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12872, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13982, 'num_bytes_recovered': 114509834, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:06.576 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:11.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:11.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:11.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:11.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:11.686 INFO:teuthology.orchestra.run.vm00.stdout:163208757280 2026-03-31T21:05:11.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:11.691 INFO:teuthology.orchestra.run.vm00.stdout:227633266709 2026-03-31T21:05:11.691 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:11.694 INFO:teuthology.orchestra.run.vm00.stdout:90194313283 2026-03-31T21:05:11.694 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:11.700 INFO:teuthology.orchestra.run.vm00.stdout:206158430231 2026-03-31T21:05:11.700 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:11.917 INFO:teuthology.orchestra.run.vm00.stdout:90194313284 2026-03-31T21:05:11.919 INFO:teuthology.orchestra.run.vm00.stdout:163208757280 2026-03-31T21:05:11.923 INFO:teuthology.orchestra.run.vm00.stdout:206158430231 2026-03-31T21:05:11.935 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313283 got 90194313284 for osd.1 2026-03-31T21:05:11.935 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:11.936 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757280 got 163208757280 for osd.0 2026-03-31T21:05:11.936 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:11.940 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430231 got 206158430231 for osd.3 2026-03-31T21:05:11.940 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:11.943 INFO:teuthology.orchestra.run.vm00.stdout:227633266710 2026-03-31T21:05:11.957 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266709 got 227633266710 for osd.2 2026-03-31T21:05:11.957 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:11.957 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:12.194 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:12.194 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:12.205 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2026-03-31T21:05:11.822474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12805,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14051,"num_bytes_recovered":115386890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":306696,"kb_used_data":111184,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282808824,"statfs":{"total":289910292480,"available":289596235776,"internally_reserved":0,"allocated":113852416,"data_stored":112978562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-120,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001732"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":41662,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:11.618271+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:11.618271+0000","last_peered":"2026-03-31T21:05:11.618271+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:11.618271+0000","last_undegraded":"2026-03-31T21:05:11.618271+0000","last_fullsized":"2026-03-31T21:05:11.618271+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12805,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14049,"num_bytes_recovered":115058698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12805,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14049,"num_bytes_recovered":115058698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166551552,"data_stored":166551552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430231,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313284,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757280,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32528,"kb_used_data":3008,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94339312,"statfs":{"total":96636764160,"available":96603455488,"internally_reserved":0,"allocated":3080192,"data_stored":2786518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2306048,"data_stored":2306048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:12.205 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 41662, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:11.618271+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:11.618271+0000', 'last_peered': '2026-03-31T21:05:11.618271+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:11.618271+0000', 'last_undegraded': '2026-03-31T21:05:11.618271+0000', 'last_fullsized': '2026-03-31T21:05:11.618271+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12805, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14049, 'num_bytes_recovered': 115058698, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:12.205 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:17.206 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:17.206 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:17.206 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:17.206 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:17.319 INFO:teuthology.orchestra.run.vm00.stdout:206158430233 2026-03-31T21:05:17.320 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:17.326 INFO:teuthology.orchestra.run.vm00.stdout:90194313286 2026-03-31T21:05:17.327 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:17.327 INFO:teuthology.orchestra.run.vm00.stdout:227633266712 2026-03-31T21:05:17.327 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:17.333 INFO:teuthology.orchestra.run.vm00.stdout:163208757282 2026-03-31T21:05:17.333 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:17.538 INFO:teuthology.orchestra.run.vm00.stdout:206158430232 2026-03-31T21:05:17.557 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430233 got 206158430232 for osd.3 2026-03-31T21:05:17.595 INFO:teuthology.orchestra.run.vm00.stdout:90194313284 2026-03-31T21:05:17.597 INFO:teuthology.orchestra.run.vm00.stdout:227633266710 2026-03-31T21:05:17.608 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266712 got 227633266710 for osd.2 2026-03-31T21:05:17.609 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313286 got 90194313284 for osd.1 2026-03-31T21:05:17.642 INFO:teuthology.orchestra.run.vm00.stdout:163208757281 2026-03-31T21:05:17.654 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757282 got 163208757281 for osd.0 2026-03-31T21:05:18.558 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:18.609 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:18.609 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:18.656 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:18.784 INFO:teuthology.orchestra.run.vm00.stdout:206158430233 2026-03-31T21:05:18.804 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430233 got 206158430233 for osd.3 2026-03-31T21:05:18.804 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:18.832 INFO:teuthology.orchestra.run.vm00.stdout:227633266712 2026-03-31T21:05:18.840 INFO:teuthology.orchestra.run.vm00.stdout:90194313286 2026-03-31T21:05:18.849 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266712 got 227633266712 for osd.2 2026-03-31T21:05:18.849 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:18.855 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313286 got 90194313286 for osd.1 2026-03-31T21:05:18.855 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:18.885 INFO:teuthology.orchestra.run.vm00.stdout:163208757282 2026-03-31T21:05:18.899 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757282 got 163208757282 for osd.0 2026-03-31T21:05:18.899 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:18.899 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:19.133 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:19.133 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:19.144 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2026-03-31T21:05:17.823231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12750,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14106,"num_bytes_recovered":115837450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":306932,"kb_used_data":111420,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282808588,"statfs":{"total":289910292480,"available":289595994112,"internally_reserved":0,"allocated":114094080,"data_stored":113220226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001547"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":41772,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:17.239144+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:17.239144+0000","last_peered":"2026-03-31T21:05:17.239144+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:17.239144+0000","last_undegraded":"2026-03-31T21:05:17.239144+0000","last_fullsized":"2026-03-31T21:05:17.239144+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12750,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14104,"num_bytes_recovered":115509258,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12750,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14104,"num_bytes_recovered":115509258,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":166776832,"data_stored":166776832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430233,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313286,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757282,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":3244,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":3321856,"data_stored":3028182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2531328,"data_stored":2531328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:19.144 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 41772, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:17.239144+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:17.239144+0000', 'last_peered': '2026-03-31T21:05:17.239144+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:17.239144+0000', 'last_undegraded': '2026-03-31T21:05:17.239144+0000', 'last_fullsized': '2026-03-31T21:05:17.239144+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12750, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14104, 'num_bytes_recovered': 115509258, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:19.144 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:24.145 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:24.145 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:24.145 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:24.145 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:24.262 INFO:teuthology.orchestra.run.vm00.stdout:90194313288 2026-03-31T21:05:24.262 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:24.265 INFO:teuthology.orchestra.run.vm00.stdout:227633266714 2026-03-31T21:05:24.265 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:24.269 INFO:teuthology.orchestra.run.vm00.stdout:206158430235 2026-03-31T21:05:24.269 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:24.275 INFO:teuthology.orchestra.run.vm00.stdout:163208757284 2026-03-31T21:05:24.276 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:24.502 INFO:teuthology.orchestra.run.vm00.stdout:90194313287 2026-03-31T21:05:24.507 INFO:teuthology.orchestra.run.vm00.stdout:163208757283 2026-03-31T21:05:24.508 INFO:teuthology.orchestra.run.vm00.stdout:227633266713 2026-03-31T21:05:24.512 INFO:teuthology.orchestra.run.vm00.stdout:206158430234 2026-03-31T21:05:24.518 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313288 got 90194313287 for osd.1 2026-03-31T21:05:24.521 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266714 got 227633266713 for osd.2 2026-03-31T21:05:24.524 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757284 got 163208757283 for osd.0 2026-03-31T21:05:24.528 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430235 got 206158430234 for osd.3 2026-03-31T21:05:25.519 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:25.522 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:25.525 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:25.529 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:25.735 INFO:teuthology.orchestra.run.vm00.stdout:90194313287 2026-03-31T21:05:25.755 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313288 got 90194313287 for osd.1 2026-03-31T21:05:25.777 INFO:teuthology.orchestra.run.vm00.stdout:163208757283 2026-03-31T21:05:25.777 INFO:teuthology.orchestra.run.vm00.stdout:206158430234 2026-03-31T21:05:25.789 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430235 got 206158430234 for osd.3 2026-03-31T21:05:25.789 INFO:teuthology.orchestra.run.vm00.stdout:227633266713 2026-03-31T21:05:25.791 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757284 got 163208757283 for osd.0 2026-03-31T21:05:25.801 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266714 got 227633266713 for osd.2 2026-03-31T21:05:26.756 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:26.789 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:26.791 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:26.801 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:26.971 INFO:teuthology.orchestra.run.vm00.stdout:90194313288 2026-03-31T21:05:26.992 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313288 got 90194313288 for osd.1 2026-03-31T21:05:26.993 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:27.005 INFO:teuthology.orchestra.run.vm00.stdout:206158430236 2026-03-31T21:05:27.017 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430235 got 206158430236 for osd.3 2026-03-31T21:05:27.017 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:27.020 INFO:teuthology.orchestra.run.vm00.stdout:163208757285 2026-03-31T21:05:27.032 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757284 got 163208757285 for osd.0 2026-03-31T21:05:27.032 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:27.073 INFO:teuthology.orchestra.run.vm00.stdout:227633266714 2026-03-31T21:05:27.084 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266714 got 227633266714 for osd.2 2026-03-31T21:05:27.084 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:27.084 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:27.324 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:27.324 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:27.336 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2026-03-31T21:05:25.824295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12680,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14176,"num_bytes_recovered":116410890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":307200,"kb_used_data":111688,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282808320,"statfs":{"total":289910292480,"available":289595719680,"internally_reserved":0,"allocated":114368512,"data_stored":113494658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-125,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":41912,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:24.388392+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:24.388392+0000","last_peered":"2026-03-31T21:05:24.388392+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:24.388392+0000","last_undegraded":"2026-03-31T21:05:24.388392+0000","last_fullsized":"2026-03-31T21:05:24.388392+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12680,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14174,"num_bytes_recovered":116082698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12680,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14174,"num_bytes_recovered":116082698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167067648,"data_stored":167067648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430236,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266714,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313288,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757285,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":3512,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":3596288,"data_stored":3302614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2822144,"data_stored":2822144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:27.336 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 41912, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:24.388392+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:24.388392+0000', 'last_peered': '2026-03-31T21:05:24.388392+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:24.388392+0000', 'last_undegraded': '2026-03-31T21:05:24.388392+0000', 'last_fullsized': '2026-03-31T21:05:24.388392+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12680, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14174, 'num_bytes_recovered': 116082698, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:27.336 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:32.337 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:32.337 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:32.337 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:32.338 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:32.442 INFO:teuthology.orchestra.run.vm00.stdout:90194313291 2026-03-31T21:05:32.442 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:32.443 INFO:teuthology.orchestra.run.vm00.stdout:206158430238 2026-03-31T21:05:32.443 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:32.446 INFO:teuthology.orchestra.run.vm00.stdout:227633266717 2026-03-31T21:05:32.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:32.447 INFO:teuthology.orchestra.run.vm00.stdout:163208757287 2026-03-31T21:05:32.447 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:32.651 INFO:teuthology.orchestra.run.vm00.stdout:206158430237 2026-03-31T21:05:32.670 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430238 got 206158430237 for osd.3 2026-03-31T21:05:32.671 INFO:teuthology.orchestra.run.vm00.stdout:227633266716 2026-03-31T21:05:32.673 INFO:teuthology.orchestra.run.vm00.stdout:90194313290 2026-03-31T21:05:32.685 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266717 got 227633266716 for osd.2 2026-03-31T21:05:32.686 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313291 got 90194313290 for osd.1 2026-03-31T21:05:32.721 INFO:teuthology.orchestra.run.vm00.stdout:163208757286 2026-03-31T21:05:32.732 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757287 got 163208757286 for osd.0 2026-03-31T21:05:33.670 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:33.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:33.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:33.733 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:33.893 INFO:teuthology.orchestra.run.vm00.stdout:206158430238 2026-03-31T21:05:33.905 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430238 got 206158430238 for osd.3 2026-03-31T21:05:33.906 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:33.928 INFO:teuthology.orchestra.run.vm00.stdout:90194313291 2026-03-31T21:05:33.943 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313291 got 90194313291 for osd.1 2026-03-31T21:05:33.943 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:33.947 INFO:teuthology.orchestra.run.vm00.stdout:227633266717 2026-03-31T21:05:33.959 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266717 got 227633266717 for osd.2 2026-03-31T21:05:33.959 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:33.977 INFO:teuthology.orchestra.run.vm00.stdout:163208757287 2026-03-31T21:05:33.989 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757287 got 163208757287 for osd.0 2026-03-31T21:05:33.989 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:33.989 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:34.226 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:34.226 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:34.237 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2026-03-31T21:05:33.825394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12602,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14254,"num_bytes_recovered":117049866,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":307516,"kb_used_data":112004,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282808004,"statfs":{"total":289910292480,"available":289595396096,"internally_reserved":0,"allocated":114692096,"data_stored":113818242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001587"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":42068,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:32.348091+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:32.348091+0000","last_peered":"2026-03-31T21:05:32.348091+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:32.348091+0000","last_undegraded":"2026-03-31T21:05:32.348091+0000","last_fullsized":"2026-03-31T21:05:32.348091+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12602,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14252,"num_bytes_recovered":116721674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996781+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996781+0000","last_peered":"2026-03-31T21:04:13.996781+0000","last_clean":"2026-03-31T21:04:13.996781+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996781+0000","last_undegraded":"2026-03-31T21:04:13.996781+0000","last_fullsized":"2026-03-31T21:04:13.996781+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12602,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14252,"num_bytes_recovered":116721674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167383040,"data_stored":167383040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430238,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313291,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757287,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33348,"kb_used_data":3828,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94338492,"statfs":{"total":96636764160,"available":96602615808,"internally_reserved":0,"allocated":3919872,"data_stored":3626198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3137536,"data_stored":3137536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:34.238 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 42068, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:32.348091+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:32.348091+0000', 'last_peered': '2026-03-31T21:05:32.348091+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:32.348091+0000', 'last_undegraded': '2026-03-31T21:05:32.348091+0000', 'last_fullsized': '2026-03-31T21:05:32.348091+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12602, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14252, 'num_bytes_recovered': 116721674, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:34.238 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:39.239 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:39.239 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:39.239 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:39.239 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:39.351 INFO:teuthology.orchestra.run.vm00.stdout:90194313293 2026-03-31T21:05:39.352 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:39.352 INFO:teuthology.orchestra.run.vm00.stdout:206158430240 2026-03-31T21:05:39.352 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:39.356 INFO:teuthology.orchestra.run.vm00.stdout:163208757289 2026-03-31T21:05:39.356 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:39.362 INFO:teuthology.orchestra.run.vm00.stdout:227633266719 2026-03-31T21:05:39.362 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:39.577 INFO:teuthology.orchestra.run.vm00.stdout:163208757288 2026-03-31T21:05:39.585 INFO:teuthology.orchestra.run.vm00.stdout:206158430239 2026-03-31T21:05:39.585 INFO:teuthology.orchestra.run.vm00.stdout:90194313292 2026-03-31T21:05:39.590 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757289 got 163208757288 for osd.0 2026-03-31T21:05:39.600 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313293 got 90194313292 for osd.1 2026-03-31T21:05:39.604 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430240 got 206158430239 for osd.3 2026-03-31T21:05:39.640 INFO:teuthology.orchestra.run.vm00.stdout:227633266718 2026-03-31T21:05:39.651 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266719 got 227633266718 for osd.2 2026-03-31T21:05:40.591 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:40.601 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:40.604 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:40.653 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:40.806 INFO:teuthology.orchestra.run.vm00.stdout:163208757290 2026-03-31T21:05:40.826 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757289 got 163208757290 for osd.0 2026-03-31T21:05:40.826 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:40.857 INFO:teuthology.orchestra.run.vm00.stdout:90194313293 2026-03-31T21:05:40.858 INFO:teuthology.orchestra.run.vm00.stdout:206158430241 2026-03-31T21:05:40.870 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430240 got 206158430241 for osd.3 2026-03-31T21:05:40.870 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:40.874 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313293 got 90194313293 for osd.1 2026-03-31T21:05:40.874 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:40.892 INFO:teuthology.orchestra.run.vm00.stdout:227633266719 2026-03-31T21:05:40.903 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266719 got 227633266719 for osd.2 2026-03-31T21:05:40.904 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:40.904 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:41.134 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:41.134 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:41.145 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2026-03-31T21:05:39.826154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12533,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14323,"num_bytes_recovered":117615114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":307788,"kb_used_data":112276,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282807732,"statfs":{"total":289910292480,"available":289595117568,"internally_reserved":0,"allocated":114970624,"data_stored":114096770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":42206,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:39.389511+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:39.389511+0000","last_peered":"2026-03-31T21:05:39.389511+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:39.389511+0000","last_undegraded":"2026-03-31T21:05:39.389511+0000","last_fullsized":"2026-03-31T21:05:39.389511+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12533,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14321,"num_bytes_recovered":117286922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12533,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14321,"num_bytes_recovered":117286922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167669760,"data_stored":167669760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430241,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757290,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33620,"kb_used_data":4100,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94338220,"statfs":{"total":96636764160,"available":96602337280,"internally_reserved":0,"allocated":4198400,"data_stored":3904726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3424256,"data_stored":3424256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:41.145 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 42206, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:39.389511+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:39.389511+0000', 'last_peered': '2026-03-31T21:05:39.389511+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:39.389511+0000', 'last_undegraded': '2026-03-31T21:05:39.389511+0000', 'last_fullsized': '2026-03-31T21:05:39.389511+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12533, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14321, 'num_bytes_recovered': 117286922, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:41.145 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:46.146 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:46.146 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:46.146 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:46.146 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:46.254 INFO:teuthology.orchestra.run.vm00.stdout:163208757292 2026-03-31T21:05:46.254 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:46.256 INFO:teuthology.orchestra.run.vm00.stdout:90194313295 2026-03-31T21:05:46.256 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:46.259 INFO:teuthology.orchestra.run.vm00.stdout:227633266721 2026-03-31T21:05:46.259 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:46.267 INFO:teuthology.orchestra.run.vm00.stdout:206158430243 2026-03-31T21:05:46.267 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:46.478 INFO:teuthology.orchestra.run.vm00.stdout:163208757291 2026-03-31T21:05:46.480 INFO:teuthology.orchestra.run.vm00.stdout:206158430242 2026-03-31T21:05:46.490 INFO:teuthology.orchestra.run.vm00.stdout:227633266720 2026-03-31T21:05:46.493 INFO:teuthology.orchestra.run.vm00.stdout:90194313294 2026-03-31T21:05:46.495 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430243 got 206158430242 for osd.3 2026-03-31T21:05:46.495 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757292 got 163208757291 for osd.0 2026-03-31T21:05:46.504 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266721 got 227633266720 for osd.2 2026-03-31T21:05:46.508 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313295 got 90194313294 for osd.1 2026-03-31T21:05:47.496 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:47.496 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:47.505 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:47.509 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:47.703 INFO:teuthology.orchestra.run.vm00.stdout:206158430242 2026-03-31T21:05:47.715 INFO:teuthology.orchestra.run.vm00.stdout:163208757291 2026-03-31T21:05:47.719 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430243 got 206158430242 for osd.3 2026-03-31T21:05:47.730 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757292 got 163208757291 for osd.0 2026-03-31T21:05:47.751 INFO:teuthology.orchestra.run.vm00.stdout:227633266720 2026-03-31T21:05:47.755 INFO:teuthology.orchestra.run.vm00.stdout:90194313294 2026-03-31T21:05:47.763 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266721 got 227633266720 for osd.2 2026-03-31T21:05:47.767 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313295 got 90194313294 for osd.1 2026-03-31T21:05:48.719 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:48.730 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:48.764 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:48.768 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:48.918 INFO:teuthology.orchestra.run.vm00.stdout:206158430243 2026-03-31T21:05:48.942 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430243 got 206158430243 for osd.3 2026-03-31T21:05:48.942 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:48.987 INFO:teuthology.orchestra.run.vm00.stdout:163208757292 2026-03-31T21:05:49.002 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757292 got 163208757292 for osd.0 2026-03-31T21:05:49.002 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:49.009 INFO:teuthology.orchestra.run.vm00.stdout:90194313296 2026-03-31T21:05:49.021 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313295 got 90194313296 for osd.1 2026-03-31T21:05:49.021 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:49.028 INFO:teuthology.orchestra.run.vm00.stdout:227633266722 2026-03-31T21:05:49.040 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266721 got 227633266722 for osd.2 2026-03-31T21:05:49.040 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:49.040 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:49.276 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:49.277 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:49.287 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2026-03-31T21:05:47.827156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12466,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14390,"num_bytes_recovered":118163978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":308056,"kb_used_data":112544,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282807464,"statfs":{"total":289910292480,"available":289594843136,"internally_reserved":0,"allocated":115245056,"data_stored":114371202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-116,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":42340,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:46.235510+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:46.235510+0000","last_peered":"2026-03-31T21:05:46.235510+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:46.235510+0000","last_undegraded":"2026-03-31T21:05:46.235510+0000","last_fullsized":"2026-03-31T21:05:46.235510+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12466,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14388,"num_bytes_recovered":117835786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12466,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14388,"num_bytes_recovered":117835786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167940096,"data_stored":167940096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430243,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313296,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33888,"kb_used_data":4368,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94337952,"statfs":{"total":96636764160,"available":96602062848,"internally_reserved":0,"allocated":4472832,"data_stored":4179158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3694592,"data_stored":3694592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:49.287 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 42340, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:46.235510+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:46.235510+0000', 'last_peered': '2026-03-31T21:05:46.235510+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:46.235510+0000', 'last_undegraded': '2026-03-31T21:05:46.235510+0000', 'last_fullsized': '2026-03-31T21:05:46.235510+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12466, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14388, 'num_bytes_recovered': 117835786, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:49.287 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:05:54.289 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:05:54.289 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:05:54.289 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:05:54.289 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:05:54.412 INFO:teuthology.orchestra.run.vm00.stdout:227633266724 2026-03-31T21:05:54.412 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:54.414 INFO:teuthology.orchestra.run.vm00.stdout:90194313298 2026-03-31T21:05:54.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:54.417 INFO:teuthology.orchestra.run.vm00.stdout:206158430245 2026-03-31T21:05:54.417 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:54.422 INFO:teuthology.orchestra.run.vm00.stdout:163208757294 2026-03-31T21:05:54.422 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:54.632 INFO:teuthology.orchestra.run.vm00.stdout:227633266723 2026-03-31T21:05:54.633 INFO:teuthology.orchestra.run.vm00.stdout:206158430244 2026-03-31T21:05:54.637 INFO:teuthology.orchestra.run.vm00.stdout:90194313297 2026-03-31T21:05:54.646 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266724 got 227633266723 for osd.2 2026-03-31T21:05:54.652 INFO:teuthology.orchestra.run.vm00.stdout:163208757293 2026-03-31T21:05:54.653 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313298 got 90194313297 for osd.1 2026-03-31T21:05:54.653 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430245 got 206158430244 for osd.3 2026-03-31T21:05:54.665 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757294 got 163208757293 for osd.0 2026-03-31T21:05:55.646 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:55.654 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:55.655 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:55.666 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:55.867 INFO:teuthology.orchestra.run.vm00.stdout:227633266723 2026-03-31T21:05:55.883 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266724 got 227633266723 for osd.2 2026-03-31T21:05:55.921 INFO:teuthology.orchestra.run.vm00.stdout:163208757293 2026-03-31T21:05:55.921 INFO:teuthology.orchestra.run.vm00.stdout:206158430244 2026-03-31T21:05:55.930 INFO:teuthology.orchestra.run.vm00.stdout:90194313297 2026-03-31T21:05:55.935 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757294 got 163208757293 for osd.0 2026-03-31T21:05:55.935 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430245 got 206158430244 for osd.3 2026-03-31T21:05:55.943 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313298 got 90194313297 for osd.1 2026-03-31T21:05:56.884 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:05:56.935 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:05:56.936 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:05:56.944 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:05:57.108 INFO:teuthology.orchestra.run.vm00.stdout:227633266724 2026-03-31T21:05:57.125 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266724 got 227633266724 for osd.2 2026-03-31T21:05:57.125 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:57.155 INFO:teuthology.orchestra.run.vm00.stdout:163208757295 2026-03-31T21:05:57.155 INFO:teuthology.orchestra.run.vm00.stdout:206158430246 2026-03-31T21:05:57.172 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430245 got 206158430246 for osd.3 2026-03-31T21:05:57.172 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:57.173 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757294 got 163208757295 for osd.0 2026-03-31T21:05:57.173 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:57.195 INFO:teuthology.orchestra.run.vm00.stdout:90194313298 2026-03-31T21:05:57.207 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313298 got 90194313298 for osd.1 2026-03-31T21:05:57.207 DEBUG:teuthology.parallel:result is None 2026-03-31T21:05:57.207 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:05:57.440 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:05:57.440 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:05:57.451 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2026-03-31T21:05:55.828271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12386,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14470,"num_bytes_recovered":118819338,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":308404,"kb_used_data":112892,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282807116,"statfs":{"total":289910292480,"available":289594486784,"internally_reserved":0,"allocated":115601408,"data_stored":114727554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001588"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":42500,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:05:54.394541+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:05:54.394541+0000","last_peered":"2026-03-31T21:05:54.394541+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:05:54.394541+0000","last_undegraded":"2026-03-31T21:05:54.394541+0000","last_fullsized":"2026-03-31T21:05:54.394541+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12386,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14468,"num_bytes_recovered":118491146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12386,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14468,"num_bytes_recovered":118491146,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":168271872,"data_stored":168271872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757295,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34236,"kb_used_data":4716,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94337604,"statfs":{"total":96636764160,"available":96601706496,"internally_reserved":0,"allocated":4829184,"data_stored":4535510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4026368,"data_stored":4026368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:05:57.452 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 42500, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:05:54.394541+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:05:54.394541+0000', 'last_peered': '2026-03-31T21:05:54.394541+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:05:54.394541+0000', 'last_undegraded': '2026-03-31T21:05:54.394541+0000', 'last_fullsized': '2026-03-31T21:05:54.394541+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12386, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14468, 'num_bytes_recovered': 118491146, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:05:57.452 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:06:02.452 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:06:02.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:06:02.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:06:02.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:06:02.555 INFO:teuthology.orchestra.run.vm00.stdout:227633266727 2026-03-31T21:06:02.556 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:02.556 INFO:teuthology.orchestra.run.vm00.stdout:163208757297 2026-03-31T21:06:02.556 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:02.561 INFO:teuthology.orchestra.run.vm00.stdout:90194313301 2026-03-31T21:06:02.561 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:02.756 INFO:teuthology.orchestra.run.vm00.stdout:206158430248 2026-03-31T21:06:02.756 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:02.766 INFO:teuthology.orchestra.run.vm00.stdout:163208757296 2026-03-31T21:06:02.774 INFO:teuthology.orchestra.run.vm00.stdout:227633266726 2026-03-31T21:06:02.783 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757297 got 163208757296 for osd.0 2026-03-31T21:06:02.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266727 got 227633266726 for osd.2 2026-03-31T21:06:02.818 INFO:teuthology.orchestra.run.vm00.stdout:90194313300 2026-03-31T21:06:02.830 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313301 got 90194313300 for osd.1 2026-03-31T21:06:02.962 INFO:teuthology.orchestra.run.vm00.stdout:206158430247 2026-03-31T21:06:02.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430248 got 206158430247 for osd.3 2026-03-31T21:06:03.784 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:03.794 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:03.831 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:03.973 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:03.997 INFO:teuthology.orchestra.run.vm00.stdout:163208757296 2026-03-31T21:06:04.012 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757297 got 163208757296 for osd.0 2026-03-31T21:06:04.043 INFO:teuthology.orchestra.run.vm00.stdout:227633266727 2026-03-31T21:06:04.055 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266727 got 227633266727 for osd.2 2026-03-31T21:06:04.055 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:04.087 INFO:teuthology.orchestra.run.vm00.stdout:90194313302 2026-03-31T21:06:04.105 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313301 got 90194313302 for osd.1 2026-03-31T21:06:04.105 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:04.202 INFO:teuthology.orchestra.run.vm00.stdout:206158430248 2026-03-31T21:06:04.215 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430248 got 206158430248 for osd.3 2026-03-31T21:06:04.215 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:05.013 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:05.210 INFO:teuthology.orchestra.run.vm00.stdout:163208757297 2026-03-31T21:06:05.223 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757297 got 163208757297 for osd.0 2026-03-31T21:06:05.223 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:05.223 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:06:05.430 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:06:05.430 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:06:05.442 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2026-03-31T21:06:03.829217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12305,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14551,"num_bytes_recovered":119482890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":308716,"kb_used_data":113204,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282806804,"statfs":{"total":289910292480,"available":289594167296,"internally_reserved":0,"allocated":115920896,"data_stored":115047042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-130,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001530"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":42663,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:06:02.668898+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:06:02.668898+0000","last_peered":"2026-03-31T21:06:02.668898+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:06:02.668898+0000","last_undegraded":"2026-03-31T21:06:02.668898+0000","last_fullsized":"2026-03-31T21:06:02.668898+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12305,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14549,"num_bytes_recovered":119154698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12305,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14549,"num_bytes_recovered":119154698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":168591360,"data_stored":168591360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430248,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313302,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757297,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34548,"kb_used_data":5028,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94337292,"statfs":{"total":96636764160,"available":96601387008,"internally_reserved":0,"allocated":5148672,"data_stored":4854998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4345856,"data_stored":4345856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:06:05.442 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 42663, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:06:02.668898+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:06:02.668898+0000', 'last_peered': '2026-03-31T21:06:02.668898+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:06:02.668898+0000', 'last_undegraded': '2026-03-31T21:06:02.668898+0000', 'last_fullsized': '2026-03-31T21:06:02.668898+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12305, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14549, 'num_bytes_recovered': 119154698, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:06:05.443 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:06:10.444 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:06:10.444 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:06:10.444 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:06:10.444 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:06:10.552 INFO:teuthology.orchestra.run.vm00.stdout:163208757300 2026-03-31T21:06:10.553 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:10.554 INFO:teuthology.orchestra.run.vm00.stdout:227633266729 2026-03-31T21:06:10.554 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:10.557 INFO:teuthology.orchestra.run.vm00.stdout:206158430251 2026-03-31T21:06:10.557 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:10.727 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:06:10.724+0000 7f24eaffd640 0 --2- 192.168.123.100:0/4196073074 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f24ec12e990 0x7f24ec12ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:06:10.751 INFO:teuthology.orchestra.run.vm00.stdout:90194313305 2026-03-31T21:06:10.752 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:10.768 INFO:teuthology.orchestra.run.vm00.stdout:163208757299 2026-03-31T21:06:10.783 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757300 got 163208757299 for osd.0 2026-03-31T21:06:10.816 INFO:teuthology.orchestra.run.vm00.stdout:227633266728 2026-03-31T21:06:10.828 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266729 got 227633266728 for osd.2 2026-03-31T21:06:10.879 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:06:10.876+0000 7fe3ceffd640 0 --2- 192.168.123.100:0/2272914127 >> v2:192.168.123.100:3300/0 conn(0x7fe3d007f640 0x7fe3d0173fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:06:10.954 INFO:teuthology.orchestra.run.vm00.stdout:90194313304 2026-03-31T21:06:10.968 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313305 got 90194313304 for osd.1 2026-03-31T21:06:11.002 INFO:teuthology.orchestra.run.vm00.stdout:206158430250 2026-03-31T21:06:11.014 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430251 got 206158430250 for osd.3 2026-03-31T21:06:11.784 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:11.829 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:11.969 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:11.989 INFO:teuthology.orchestra.run.vm00.stdout:163208757299 2026-03-31T21:06:12.003 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757300 got 163208757299 for osd.0 2026-03-31T21:06:12.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:12.031 INFO:teuthology.orchestra.run.vm00.stdout:227633266728 2026-03-31T21:06:12.047 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266729 got 227633266728 for osd.2 2026-03-31T21:06:12.177 INFO:teuthology.orchestra.run.vm00.stdout:90194313305 2026-03-31T21:06:12.190 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313305 got 90194313305 for osd.1 2026-03-31T21:06:12.191 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:12.234 INFO:teuthology.orchestra.run.vm00.stdout:206158430251 2026-03-31T21:06:12.245 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430251 got 206158430251 for osd.3 2026-03-31T21:06:12.245 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:13.003 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:13.047 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:13.203 INFO:teuthology.orchestra.run.vm00.stdout:163208757300 2026-03-31T21:06:13.214 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757300 got 163208757300 for osd.0 2026-03-31T21:06:13.214 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:13.242 INFO:teuthology.orchestra.run.vm00.stdout:227633266730 2026-03-31T21:06:13.253 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266729 got 227633266730 for osd.2 2026-03-31T21:06:13.253 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:13.253 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:06:13.486 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:06:13.486 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:06:13.497 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2026-03-31T21:06:11.830147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12231,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14625,"num_bytes_recovered":120089098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":309016,"kb_used_data":113504,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282806504,"statfs":{"total":289910292480,"available":289593860096,"internally_reserved":0,"allocated":116228096,"data_stored":115354242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-106,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001398"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":42811,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:06:10.453476+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:06:10.453476+0000","last_peered":"2026-03-31T21:06:10.453476+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:06:10.453476+0000","last_undegraded":"2026-03-31T21:06:10.453476+0000","last_fullsized":"2026-03-31T21:06:10.453476+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12231,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14623,"num_bytes_recovered":119760906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12231,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14623,"num_bytes_recovered":119760906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":168902656,"data_stored":168902656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430251,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313305,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757300,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34848,"kb_used_data":5328,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94336992,"statfs":{"total":96636764160,"available":96601079808,"internally_reserved":0,"allocated":5455872,"data_stored":5162198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4657152,"data_stored":4657152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:06:13.497 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 42811, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:06:10.453476+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:06:10.453476+0000', 'last_peered': '2026-03-31T21:06:10.453476+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:06:10.453476+0000', 'last_undegraded': '2026-03-31T21:06:10.453476+0000', 'last_fullsized': '2026-03-31T21:06:10.453476+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12231, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14623, 'num_bytes_recovered': 119760906, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:06:13.497 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:06:18.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:06:18.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:06:18.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:06:18.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:06:18.604 INFO:teuthology.orchestra.run.vm00.stdout:206158430253 2026-03-31T21:06:18.604 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:18.604 INFO:teuthology.orchestra.run.vm00.stdout:90194313308 2026-03-31T21:06:18.604 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:18.606 INFO:teuthology.orchestra.run.vm00.stdout:163208757302 2026-03-31T21:06:18.606 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:18.611 INFO:teuthology.orchestra.run.vm00.stdout:227633266732 2026-03-31T21:06:18.612 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:18.812 INFO:teuthology.orchestra.run.vm00.stdout:206158430252 2026-03-31T21:06:18.823 INFO:teuthology.orchestra.run.vm00.stdout:90194313307 2026-03-31T21:06:18.825 INFO:teuthology.orchestra.run.vm00.stdout:163208757301 2026-03-31T21:06:18.829 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430253 got 206158430252 for osd.3 2026-03-31T21:06:18.839 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313308 got 90194313307 for osd.1 2026-03-31T21:06:18.843 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757302 got 163208757301 for osd.0 2026-03-31T21:06:18.873 INFO:teuthology.orchestra.run.vm00.stdout:227633266731 2026-03-31T21:06:18.884 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266732 got 227633266731 for osd.2 2026-03-31T21:06:19.830 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:19.841 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:19.844 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:19.885 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:20.060 INFO:teuthology.orchestra.run.vm00.stdout:206158430252 2026-03-31T21:06:20.075 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430253 got 206158430252 for osd.3 2026-03-31T21:06:20.104 INFO:teuthology.orchestra.run.vm00.stdout:90194313308 2026-03-31T21:06:20.107 INFO:teuthology.orchestra.run.vm00.stdout:163208757303 2026-03-31T21:06:20.117 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313308 got 90194313308 for osd.1 2026-03-31T21:06:20.118 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:20.124 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757302 got 163208757303 for osd.0 2026-03-31T21:06:20.124 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:20.157 INFO:teuthology.orchestra.run.vm00.stdout:227633266732 2026-03-31T21:06:20.169 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266732 got 227633266732 for osd.2 2026-03-31T21:06:20.169 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:21.076 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:21.275 INFO:teuthology.orchestra.run.vm00.stdout:206158430254 2026-03-31T21:06:21.289 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430253 got 206158430254 for osd.3 2026-03-31T21:06:21.289 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:21.289 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:06:21.482 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:06:21.482 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:06:21.494 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2026-03-31T21:06:19.831184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12144,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14712,"num_bytes_recovered":120801802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":309372,"kb_used_data":113860,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282806148,"statfs":{"total":289910292480,"available":289593495552,"internally_reserved":0,"allocated":116592640,"data_stored":115718786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001481"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":42985,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:06:19.348357+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:06:19.348357+0000","last_peered":"2026-03-31T21:06:19.348357+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:06:19.348357+0000","last_undegraded":"2026-03-31T21:06:19.348357+0000","last_fullsized":"2026-03-31T21:06:19.348357+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12144,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14710,"num_bytes_recovered":120473610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12144,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14710,"num_bytes_recovered":120473610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169263104,"data_stored":169263104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430254,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35204,"kb_used_data":5684,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94336636,"statfs":{"total":96636764160,"available":96600715264,"internally_reserved":0,"allocated":5820416,"data_stored":5526742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5017600,"data_stored":5017600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:06:21.494 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 42985, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:06:19.348357+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:06:19.348357+0000', 'last_peered': '2026-03-31T21:06:19.348357+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:06:19.348357+0000', 'last_undegraded': '2026-03-31T21:06:19.348357+0000', 'last_fullsized': '2026-03-31T21:06:19.348357+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12144, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14710, 'num_bytes_recovered': 120473610, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:06:21.494 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:06:26.494 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:06:26.495 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:06:26.495 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:06:26.495 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:06:26.604 INFO:teuthology.orchestra.run.vm00.stdout:163208757305 2026-03-31T21:06:26.604 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:26.606 INFO:teuthology.orchestra.run.vm00.stdout:90194313310 2026-03-31T21:06:26.606 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:26.611 INFO:teuthology.orchestra.run.vm00.stdout:227633266734 2026-03-31T21:06:26.612 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:26.619 INFO:teuthology.orchestra.run.vm00.stdout:206158430256 2026-03-31T21:06:26.619 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:26.826 INFO:teuthology.orchestra.run.vm00.stdout:163208757304 2026-03-31T21:06:26.835 INFO:teuthology.orchestra.run.vm00.stdout:227633266733 2026-03-31T21:06:26.838 INFO:teuthology.orchestra.run.vm00.stdout:90194313309 2026-03-31T21:06:26.845 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757305 got 163208757304 for osd.0 2026-03-31T21:06:26.852 INFO:teuthology.orchestra.run.vm00.stdout:206158430255 2026-03-31T21:06:26.852 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266734 got 227633266733 for osd.2 2026-03-31T21:06:26.853 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313310 got 90194313309 for osd.1 2026-03-31T21:06:26.863 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430256 got 206158430255 for osd.3 2026-03-31T21:06:27.846 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:27.853 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:27.853 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:27.864 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:28.064 INFO:teuthology.orchestra.run.vm00.stdout:163208757304 2026-03-31T21:06:28.081 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757305 got 163208757304 for osd.0 2026-03-31T21:06:28.106 INFO:teuthology.orchestra.run.vm00.stdout:206158430255 2026-03-31T21:06:28.107 INFO:teuthology.orchestra.run.vm00.stdout:90194313309 2026-03-31T21:06:28.110 INFO:teuthology.orchestra.run.vm00.stdout:227633266733 2026-03-31T21:06:28.120 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430256 got 206158430255 for osd.3 2026-03-31T21:06:28.121 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313310 got 90194313309 for osd.1 2026-03-31T21:06:28.129 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266734 got 227633266733 for osd.2 2026-03-31T21:06:29.081 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:29.121 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:29.122 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:29.129 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:29.298 INFO:teuthology.orchestra.run.vm00.stdout:163208757304 2026-03-31T21:06:29.310 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757305 got 163208757304 for osd.0 2026-03-31T21:06:29.337 INFO:teuthology.orchestra.run.vm00.stdout:206158430255 2026-03-31T21:06:29.346 INFO:teuthology.orchestra.run.vm00.stdout:90194313309 2026-03-31T21:06:29.349 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430256 got 206158430255 for osd.3 2026-03-31T21:06:29.361 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313310 got 90194313309 for osd.1 2026-03-31T21:06:29.415 INFO:teuthology.orchestra.run.vm00.stdout:227633266733 2026-03-31T21:06:29.427 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266734 got 227633266733 for osd.2 2026-03-31T21:06:30.311 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:30.349 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:30.362 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:30.427 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:30.516 INFO:teuthology.orchestra.run.vm00.stdout:163208757306 2026-03-31T21:06:30.537 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757305 got 163208757306 for osd.0 2026-03-31T21:06:30.538 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:30.565 INFO:teuthology.orchestra.run.vm00.stdout:206158430257 2026-03-31T21:06:30.577 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430256 got 206158430257 for osd.3 2026-03-31T21:06:30.577 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:30.610 INFO:teuthology.orchestra.run.vm00.stdout:90194313311 2026-03-31T21:06:30.621 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313310 got 90194313311 for osd.1 2026-03-31T21:06:30.621 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:30.682 INFO:teuthology.orchestra.run.vm00.stdout:227633266735 2026-03-31T21:06:30.694 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266734 got 227633266735 for osd.2 2026-03-31T21:06:30.694 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:30.694 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:06:30.881 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:06:30.881 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:06:30.893 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2026-03-31T21:06:29.832456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12046,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14810,"num_bytes_recovered":121604618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":309756,"kb_used_data":114244,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282805764,"statfs":{"total":289910292480,"available":289593102336,"internally_reserved":0,"allocated":116985856,"data_stored":116112002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":43181,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:06:29.351977+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:06:29.351977+0000","last_peered":"2026-03-31T21:06:29.351977+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:06:29.351977+0000","last_undegraded":"2026-03-31T21:06:29.351977+0000","last_fullsized":"2026-03-31T21:06:29.351977+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12046,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14808,"num_bytes_recovered":121276426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12046,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14808,"num_bytes_recovered":121276426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169664512,"data_stored":169664512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430257,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266735,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757306,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35588,"kb_used_data":6068,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94336252,"statfs":{"total":96636764160,"available":96600322048,"internally_reserved":0,"allocated":6213632,"data_stored":5919958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5419008,"data_stored":5419008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:06:30.893 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 43181, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:06:29.351977+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:06:29.351977+0000', 'last_peered': '2026-03-31T21:06:29.351977+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:06:29.351977+0000', 'last_undegraded': '2026-03-31T21:06:29.351977+0000', 'last_fullsized': '2026-03-31T21:06:29.351977+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 12046, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14808, 'num_bytes_recovered': 121276426, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:06:30.893 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:06:35.895 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:06:35.895 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:06:35.895 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:06:35.895 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:06:36.009 INFO:teuthology.orchestra.run.vm00.stdout:90194313313 2026-03-31T21:06:36.009 INFO:teuthology.orchestra.run.vm00.stdout:206158430259 2026-03-31T21:06:36.009 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:36.009 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:36.010 INFO:teuthology.orchestra.run.vm00.stdout:163208757308 2026-03-31T21:06:36.010 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:36.016 INFO:teuthology.orchestra.run.vm00.stdout:227633266737 2026-03-31T21:06:36.017 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:36.230 INFO:teuthology.orchestra.run.vm00.stdout:206158430258 2026-03-31T21:06:36.234 INFO:teuthology.orchestra.run.vm00.stdout:90194313312 2026-03-31T21:06:36.253 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430259 got 206158430258 for osd.3 2026-03-31T21:06:36.253 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313313 got 90194313312 for osd.1 2026-03-31T21:06:36.281 INFO:teuthology.orchestra.run.vm00.stdout:163208757307 2026-03-31T21:06:36.283 INFO:teuthology.orchestra.run.vm00.stdout:227633266736 2026-03-31T21:06:36.294 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266737 got 227633266736 for osd.2 2026-03-31T21:06:36.296 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757308 got 163208757307 for osd.0 2026-03-31T21:06:37.254 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:37.254 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:37.295 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:37.297 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:37.492 INFO:teuthology.orchestra.run.vm00.stdout:90194313312 2026-03-31T21:06:37.496 INFO:teuthology.orchestra.run.vm00.stdout:206158430258 2026-03-31T21:06:37.507 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313313 got 90194313312 for osd.1 2026-03-31T21:06:37.515 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430259 got 206158430258 for osd.3 2026-03-31T21:06:37.526 INFO:teuthology.orchestra.run.vm00.stdout:227633266736 2026-03-31T21:06:37.532 INFO:teuthology.orchestra.run.vm00.stdout:163208757307 2026-03-31T21:06:37.541 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266737 got 227633266736 for osd.2 2026-03-31T21:06:37.548 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757308 got 163208757307 for osd.0 2026-03-31T21:06:38.507 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:38.515 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:38.542 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:38.549 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:38.764 INFO:teuthology.orchestra.run.vm00.stdout:90194313314 2026-03-31T21:06:38.780 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313313 got 90194313314 for osd.1 2026-03-31T21:06:38.780 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:38.781 INFO:teuthology.orchestra.run.vm00.stdout:206158430259 2026-03-31T21:06:38.791 INFO:teuthology.orchestra.run.vm00.stdout:227633266738 2026-03-31T21:06:38.799 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430259 got 206158430259 for osd.3 2026-03-31T21:06:38.799 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:38.801 INFO:teuthology.orchestra.run.vm00.stdout:163208757308 2026-03-31T21:06:38.805 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266737 got 227633266738 for osd.2 2026-03-31T21:06:38.805 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:38.813 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757308 got 163208757308 for osd.0 2026-03-31T21:06:38.813 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:38.813 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:06:39.049 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:06:39.049 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:06:39.061 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2026-03-31T21:06:37.833598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11981,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14875,"num_bytes_recovered":122137098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":309988,"kb_used_data":114476,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282805532,"statfs":{"total":289910292480,"available":289592864768,"internally_reserved":0,"allocated":117223424,"data_stored":116349570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-114,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":43311,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:06:35.981405+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:06:35.981405+0000","last_peered":"2026-03-31T21:06:35.981405+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:06:35.981405+0000","last_undegraded":"2026-03-31T21:06:35.981405+0000","last_fullsized":"2026-03-31T21:06:35.981405+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11981,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14873,"num_bytes_recovered":121808906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11981,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14873,"num_bytes_recovered":121808906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":169926656,"data_stored":169926656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313314,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757308,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35820,"kb_used_data":6300,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94336020,"statfs":{"total":96636764160,"available":96600084480,"internally_reserved":0,"allocated":6451200,"data_stored":6157526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5681152,"data_stored":5681152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:06:39.061 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 43311, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:06:35.981405+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:06:35.981405+0000', 'last_peered': '2026-03-31T21:06:35.981405+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:06:35.981405+0000', 'last_undegraded': '2026-03-31T21:06:35.981405+0000', 'last_fullsized': '2026-03-31T21:06:35.981405+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11981, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14873, 'num_bytes_recovered': 121808906, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:06:39.061 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:06:44.062 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:06:44.063 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:06:44.063 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:06:44.063 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:06:44.173 INFO:teuthology.orchestra.run.vm00.stdout:206158430261 2026-03-31T21:06:44.173 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:44.175 INFO:teuthology.orchestra.run.vm00.stdout:227633266740 2026-03-31T21:06:44.175 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:44.179 INFO:teuthology.orchestra.run.vm00.stdout:90194313316 2026-03-31T21:06:44.180 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:44.377 INFO:teuthology.orchestra.run.vm00.stdout:163208757310 2026-03-31T21:06:44.377 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:44.393 INFO:teuthology.orchestra.run.vm00.stdout:206158430260 2026-03-31T21:06:44.396 INFO:teuthology.orchestra.run.vm00.stdout:227633266739 2026-03-31T21:06:44.406 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430261 got 206158430260 for osd.3 2026-03-31T21:06:44.413 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266740 got 227633266739 for osd.2 2026-03-31T21:06:44.433 INFO:teuthology.orchestra.run.vm00.stdout:90194313315 2026-03-31T21:06:44.445 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313316 got 90194313315 for osd.1 2026-03-31T21:06:44.586 INFO:teuthology.orchestra.run.vm00.stdout:163208757309 2026-03-31T21:06:44.597 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757310 got 163208757309 for osd.0 2026-03-31T21:06:45.407 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:45.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:45.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:45.598 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:45.619 INFO:teuthology.orchestra.run.vm00.stdout:206158430260 2026-03-31T21:06:45.641 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430261 got 206158430260 for osd.3 2026-03-31T21:06:45.664 INFO:teuthology.orchestra.run.vm00.stdout:227633266739 2026-03-31T21:06:45.664 INFO:teuthology.orchestra.run.vm00.stdout:90194313315 2026-03-31T21:06:45.677 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313316 got 90194313315 for osd.1 2026-03-31T21:06:45.678 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266740 got 227633266739 for osd.2 2026-03-31T21:06:45.815 INFO:teuthology.orchestra.run.vm00.stdout:163208757309 2026-03-31T21:06:45.826 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757310 got 163208757309 for osd.0 2026-03-31T21:06:46.642 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:46.678 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:46.679 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:46.827 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:46.849 INFO:teuthology.orchestra.run.vm00.stdout:206158430262 2026-03-31T21:06:46.861 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430261 got 206158430262 for osd.3 2026-03-31T21:06:46.862 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:46.895 INFO:teuthology.orchestra.run.vm00.stdout:227633266740 2026-03-31T21:06:46.908 INFO:teuthology.orchestra.run.vm00.stdout:90194313316 2026-03-31T21:06:46.914 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266740 got 227633266740 for osd.2 2026-03-31T21:06:46.914 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:46.923 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313316 got 90194313316 for osd.1 2026-03-31T21:06:46.923 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:47.037 INFO:teuthology.orchestra.run.vm00.stdout:163208757311 2026-03-31T21:06:47.049 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757310 got 163208757311 for osd.0 2026-03-31T21:06:47.049 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:47.049 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:06:47.246 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:06:47.246 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:06:47.257 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2026-03-31T21:06:45.834617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11900,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14956,"num_bytes_recovered":122800650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":310344,"kb_used_data":114832,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282805176,"statfs":{"total":289910292480,"available":289592500224,"internally_reserved":0,"allocated":117587968,"data_stored":116714114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001600"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":43473,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:06:44.316455+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:06:44.316455+0000","last_peered":"2026-03-31T21:06:44.316455+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:06:44.316455+0000","last_undegraded":"2026-03-31T21:06:44.316455+0000","last_fullsized":"2026-03-31T21:06:44.316455+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14954,"num_bytes_recovered":122472458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14954,"num_bytes_recovered":122472458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":170266624,"data_stored":170266624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757311,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36176,"kb_used_data":6656,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94335664,"statfs":{"total":96636764160,"available":96599719936,"internally_reserved":0,"allocated":6815744,"data_stored":6522070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6021120,"data_stored":6021120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:06:47.257 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 43473, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:06:44.316455+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:06:44.316455+0000', 'last_peered': '2026-03-31T21:06:44.316455+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:06:44.316455+0000', 'last_undegraded': '2026-03-31T21:06:44.316455+0000', 'last_fullsized': '2026-03-31T21:06:44.316455+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11900, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 14954, 'num_bytes_recovered': 122472458, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:06:47.257 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:06:52.258 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:06:52.258 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:06:52.258 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:06:52.258 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:06:52.371 INFO:teuthology.orchestra.run.vm00.stdout:163208757313 2026-03-31T21:06:52.371 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:52.374 INFO:teuthology.orchestra.run.vm00.stdout:227633266743 2026-03-31T21:06:52.374 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:52.377 INFO:teuthology.orchestra.run.vm00.stdout:206158430264 2026-03-31T21:06:52.377 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:52.382 INFO:teuthology.orchestra.run.vm00.stdout:90194313318 2026-03-31T21:06:52.383 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:52.594 INFO:teuthology.orchestra.run.vm00.stdout:163208757312 2026-03-31T21:06:52.594 INFO:teuthology.orchestra.run.vm00.stdout:206158430263 2026-03-31T21:06:52.605 INFO:teuthology.orchestra.run.vm00.stdout:90194313317 2026-03-31T21:06:52.606 INFO:teuthology.orchestra.run.vm00.stdout:227633266742 2026-03-31T21:06:52.607 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430264 got 206158430263 for osd.3 2026-03-31T21:06:52.608 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757313 got 163208757312 for osd.0 2026-03-31T21:06:52.618 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266743 got 227633266742 for osd.2 2026-03-31T21:06:52.618 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313318 got 90194313317 for osd.1 2026-03-31T21:06:53.608 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:53.608 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:53.618 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:06:53.618 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:53.826 INFO:teuthology.orchestra.run.vm00.stdout:163208757312 2026-03-31T21:06:53.836 INFO:teuthology.orchestra.run.vm00.stdout:206158430263 2026-03-31T21:06:53.843 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757313 got 163208757312 for osd.0 2026-03-31T21:06:53.851 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430264 got 206158430263 for osd.3 2026-03-31T21:06:53.867 INFO:teuthology.orchestra.run.vm00.stdout:90194313317 2026-03-31T21:06:53.878 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313318 got 90194313317 for osd.1 2026-03-31T21:06:53.940 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:06:53.938+0000 7f04fb7fe640 0 --2- 192.168.123.100:0/416307211 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f04fc1716a0 0x7f04fc18c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:06:54.061 INFO:teuthology.orchestra.run.vm00.stdout:227633266743 2026-03-31T21:06:54.073 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266743 got 227633266743 for osd.2 2026-03-31T21:06:54.073 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:54.845 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:06:54.852 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:06:54.880 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:06:55.056 INFO:teuthology.orchestra.run.vm00.stdout:163208757313 2026-03-31T21:06:55.073 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757313 got 163208757313 for osd.0 2026-03-31T21:06:55.073 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:55.091 INFO:teuthology.orchestra.run.vm00.stdout:90194313319 2026-03-31T21:06:55.099 INFO:teuthology.orchestra.run.vm00.stdout:206158430264 2026-03-31T21:06:55.107 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313318 got 90194313319 for osd.1 2026-03-31T21:06:55.107 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:55.110 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430264 got 206158430264 for osd.3 2026-03-31T21:06:55.110 DEBUG:teuthology.parallel:result is None 2026-03-31T21:06:55.110 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:06:55.352 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:06:55.352 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:06:55.368 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2026-03-31T21:06:53.835768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11822,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15034,"num_bytes_recovered":123439626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":310660,"kb_used_data":115148,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282804860,"statfs":{"total":289910292480,"available":289592176640,"internally_reserved":0,"allocated":117911552,"data_stored":117037698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001648"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":43629,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:06:52.273190+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:06:52.273190+0000","last_peered":"2026-03-31T21:06:52.273190+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:06:52.273190+0000","last_undegraded":"2026-03-31T21:06:52.273190+0000","last_fullsized":"2026-03-31T21:06:52.273190+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11822,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15032,"num_bytes_recovered":123111434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11822,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15032,"num_bytes_recovered":123111434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":170577920,"data_stored":170577920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430264,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266743,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757313,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":6972,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":7139328,"data_stored":6845654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6332416,"data_stored":6332416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:06:55.368 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 43629, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:06:52.273190+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:06:52.273190+0000', 'last_peered': '2026-03-31T21:06:52.273190+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:06:52.273190+0000', 'last_undegraded': '2026-03-31T21:06:52.273190+0000', 'last_fullsized': '2026-03-31T21:06:52.273190+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11822, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15032, 'num_bytes_recovered': 123111434, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:06:55.369 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:00.369 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:00.370 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:00.370 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:00.370 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:00.480 INFO:teuthology.orchestra.run.vm00.stdout:163208757316 2026-03-31T21:07:00.480 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:00.482 INFO:teuthology.orchestra.run.vm00.stdout:227633266745 2026-03-31T21:07:00.482 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:00.484 INFO:teuthology.orchestra.run.vm00.stdout:90194313321 2026-03-31T21:07:00.484 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:00.684 INFO:teuthology.orchestra.run.vm00.stdout:206158430267 2026-03-31T21:07:00.684 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:00.686 INFO:teuthology.orchestra.run.vm00.stdout:163208757315 2026-03-31T21:07:00.690 INFO:teuthology.orchestra.run.vm00.stdout:227633266744 2026-03-31T21:07:00.697 INFO:teuthology.orchestra.run.vm00.stdout:90194313320 2026-03-31T21:07:00.703 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757316 got 163208757315 for osd.0 2026-03-31T21:07:00.706 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266745 got 227633266744 for osd.2 2026-03-31T21:07:00.712 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313321 got 90194313320 for osd.1 2026-03-31T21:07:00.879 INFO:teuthology.orchestra.run.vm00.stdout:206158430266 2026-03-31T21:07:00.890 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430267 got 206158430266 for osd.3 2026-03-31T21:07:01.703 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:01.706 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:01.712 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:01.891 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:01.908 INFO:teuthology.orchestra.run.vm00.stdout:163208757315 2026-03-31T21:07:01.926 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757316 got 163208757315 for osd.0 2026-03-31T21:07:01.941 INFO:teuthology.orchestra.run.vm00.stdout:90194313320 2026-03-31T21:07:01.945 INFO:teuthology.orchestra.run.vm00.stdout:227633266744 2026-03-31T21:07:01.953 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313321 got 90194313320 for osd.1 2026-03-31T21:07:01.960 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266745 got 227633266744 for osd.2 2026-03-31T21:07:02.093 INFO:teuthology.orchestra.run.vm00.stdout:206158430267 2026-03-31T21:07:02.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430267 got 206158430267 for osd.3 2026-03-31T21:07:02.104 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:02.927 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:02.954 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:02.961 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:03.136 INFO:teuthology.orchestra.run.vm00.stdout:163208757316 2026-03-31T21:07:03.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757316 got 163208757316 for osd.0 2026-03-31T21:07:03.148 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:03.167 INFO:teuthology.orchestra.run.vm00.stdout:90194313321 2026-03-31T21:07:03.174 INFO:teuthology.orchestra.run.vm00.stdout:227633266746 2026-03-31T21:07:03.178 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313321 got 90194313321 for osd.1 2026-03-31T21:07:03.178 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:03.186 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266745 got 227633266746 for osd.2 2026-03-31T21:07:03.186 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:03.186 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:03.410 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:03.410 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:03.420 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2026-03-31T21:07:01.836727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11740,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15116,"num_bytes_recovered":124111370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":310976,"kb_used_data":115464,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282804544,"statfs":{"total":289910292480,"available":289591853056,"internally_reserved":0,"allocated":118235136,"data_stored":117361282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-110,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":901120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":43793,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:00.643901+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:00.643901+0000","last_peered":"2026-03-31T21:07:00.643901+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:00.643901+0000","last_undegraded":"2026-03-31T21:07:00.643901+0000","last_fullsized":"2026-03-31T21:07:00.643901+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11740,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15114,"num_bytes_recovered":123783178,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11740,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15114,"num_bytes_recovered":123783178,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":170905600,"data_stored":170905600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430267,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757316,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36808,"kb_used_data":7288,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94335032,"statfs":{"total":96636764160,"available":96599072768,"internally_reserved":0,"allocated":7462912,"data_stored":7169238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6660096,"data_stored":6660096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:03.420 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 43793, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:00.643901+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:00.643901+0000', 'last_peered': '2026-03-31T21:07:00.643901+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:00.643901+0000', 'last_undegraded': '2026-03-31T21:07:00.643901+0000', 'last_fullsized': '2026-03-31T21:07:00.643901+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11740, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15114, 'num_bytes_recovered': 123783178, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:03.420 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:08.421 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:08.421 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:08.422 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:08.422 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:08.528 INFO:teuthology.orchestra.run.vm00.stdout:163208757318 2026-03-31T21:07:08.528 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:08.529 INFO:teuthology.orchestra.run.vm00.stdout:90194313324 2026-03-31T21:07:08.529 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:08.531 INFO:teuthology.orchestra.run.vm00.stdout:206158430269 2026-03-31T21:07:08.532 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:08.537 INFO:teuthology.orchestra.run.vm00.stdout:227633266748 2026-03-31T21:07:08.537 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:08.745 INFO:teuthology.orchestra.run.vm00.stdout:206158430268 2026-03-31T21:07:08.748 INFO:teuthology.orchestra.run.vm00.stdout:163208757317 2026-03-31T21:07:08.749 INFO:teuthology.orchestra.run.vm00.stdout:90194313323 2026-03-31T21:07:08.760 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430269 got 206158430268 for osd.3 2026-03-31T21:07:08.760 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313324 got 90194313323 for osd.1 2026-03-31T21:07:08.765 INFO:teuthology.orchestra.run.vm00.stdout:227633266747 2026-03-31T21:07:08.767 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757318 got 163208757317 for osd.0 2026-03-31T21:07:08.776 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266748 got 227633266747 for osd.2 2026-03-31T21:07:09.760 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:09.760 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:09.767 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:09.777 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:09.964 INFO:teuthology.orchestra.run.vm00.stdout:206158430268 2026-03-31T21:07:09.972 INFO:teuthology.orchestra.run.vm00.stdout:90194313323 2026-03-31T21:07:09.980 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430269 got 206158430268 for osd.3 2026-03-31T21:07:09.987 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313324 got 90194313323 for osd.1 2026-03-31T21:07:10.009 INFO:teuthology.orchestra.run.vm00.stdout:163208757317 2026-03-31T21:07:10.022 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757318 got 163208757317 for osd.0 2026-03-31T21:07:10.025 INFO:teuthology.orchestra.run.vm00.stdout:227633266747 2026-03-31T21:07:10.037 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266748 got 227633266747 for osd.2 2026-03-31T21:07:10.981 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:10.987 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:11.023 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:11.038 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:11.184 INFO:teuthology.orchestra.run.vm00.stdout:206158430270 2026-03-31T21:07:11.198 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430269 got 206158430270 for osd.3 2026-03-31T21:07:11.198 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:11.225 INFO:teuthology.orchestra.run.vm00.stdout:163208757319 2026-03-31T21:07:11.237 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757318 got 163208757319 for osd.0 2026-03-31T21:07:11.237 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:11.244 INFO:teuthology.orchestra.run.vm00.stdout:90194313324 2026-03-31T21:07:11.256 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313324 got 90194313324 for osd.1 2026-03-31T21:07:11.256 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:11.293 INFO:teuthology.orchestra.run.vm00.stdout:227633266748 2026-03-31T21:07:11.303 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266748 got 227633266748 for osd.2 2026-03-31T21:07:11.303 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:11.303 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:11.614 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:07:11.612+0000 7ff3d8b6f640 0 --2- 192.168.123.100:0/2144982291 >> [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] conn(0x7ff3d4114a90 0x7ff3d4108f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:07:11.733 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:11.734 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:11.745 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2026-03-31T21:07:09.837830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11654,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15202,"num_bytes_recovered":124815882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":311324,"kb_used_data":115812,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282804196,"statfs":{"total":289910292480,"available":289591496704,"internally_reserved":0,"allocated":118591488,"data_stored":117717634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":43965,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:09.395777+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:09.395777+0000","last_peered":"2026-03-31T21:07:09.395777+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:09.395777+0000","last_undegraded":"2026-03-31T21:07:09.395777+0000","last_fullsized":"2026-03-31T21:07:09.395777+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11654,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15200,"num_bytes_recovered":124487690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11654,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15200,"num_bytes_recovered":124487690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171270144,"data_stored":171270144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430270,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313324,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757319,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37156,"kb_used_data":7636,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94334684,"statfs":{"total":96636764160,"available":96598716416,"internally_reserved":0,"allocated":7819264,"data_stored":7525590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7024640,"data_stored":7024640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:11.746 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 43965, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:09.395777+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:09.395777+0000', 'last_peered': '2026-03-31T21:07:09.395777+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:09.395777+0000', 'last_undegraded': '2026-03-31T21:07:09.395777+0000', 'last_fullsized': '2026-03-31T21:07:09.395777+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11654, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15200, 'num_bytes_recovered': 124487690, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:11.746 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:16.746 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:16.746 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:16.747 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:16.747 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:16.849 INFO:teuthology.orchestra.run.vm00.stdout:206158430272 2026-03-31T21:07:16.849 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:16.855 INFO:teuthology.orchestra.run.vm00.stdout:90194313326 2026-03-31T21:07:16.855 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:16.859 INFO:teuthology.orchestra.run.vm00.stdout:227633266751 2026-03-31T21:07:16.859 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:16.865 INFO:teuthology.orchestra.run.vm00.stdout:163208757321 2026-03-31T21:07:16.866 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:17.057 INFO:teuthology.orchestra.run.vm00.stdout:206158430271 2026-03-31T21:07:17.071 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430272 got 206158430271 for osd.3 2026-03-31T21:07:17.073 INFO:teuthology.orchestra.run.vm00.stdout:163208757320 2026-03-31T21:07:17.079 INFO:teuthology.orchestra.run.vm00.stdout:90194313325 2026-03-31T21:07:17.079 INFO:teuthology.orchestra.run.vm00.stdout:227633266749 2026-03-31T21:07:17.088 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757321 got 163208757320 for osd.0 2026-03-31T21:07:17.090 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313326 got 90194313325 for osd.1 2026-03-31T21:07:17.095 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266751 got 227633266749 for osd.2 2026-03-31T21:07:18.072 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:18.088 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:18.091 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:18.096 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:18.290 INFO:teuthology.orchestra.run.vm00.stdout:206158430272 2026-03-31T21:07:18.307 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430272 got 206158430272 for osd.3 2026-03-31T21:07:18.307 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:18.321 INFO:teuthology.orchestra.run.vm00.stdout:90194313327 2026-03-31T21:07:18.330 INFO:teuthology.orchestra.run.vm00.stdout:163208757321 2026-03-31T21:07:18.331 INFO:teuthology.orchestra.run.vm00.stdout:227633266751 2026-03-31T21:07:18.333 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313326 got 90194313327 for osd.1 2026-03-31T21:07:18.333 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:18.342 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757321 got 163208757321 for osd.0 2026-03-31T21:07:18.342 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:18.344 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266751 got 227633266751 for osd.2 2026-03-31T21:07:18.344 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:18.344 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:18.575 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:18.575 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:18.586 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-03-31T21:07:17.838839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11582,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15274,"num_bytes_recovered":125405706,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":311596,"kb_used_data":116084,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282803924,"statfs":{"total":289910292480,"available":289591218176,"internally_reserved":0,"allocated":118870016,"data_stored":117996162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-121,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44109,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:16.765432+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:16.765432+0000","last_peered":"2026-03-31T21:07:16.765432+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:16.765432+0000","last_undegraded":"2026-03-31T21:07:16.765432+0000","last_fullsized":"2026-03-31T21:07:16.765432+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11582,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15272,"num_bytes_recovered":125077514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11582,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15272,"num_bytes_recovered":125077514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171560960,"data_stored":171560960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430272,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313327,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37428,"kb_used_data":7908,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94334412,"statfs":{"total":96636764160,"available":96598437888,"internally_reserved":0,"allocated":8097792,"data_stored":7804118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7315456,"data_stored":7315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:18.586 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44109, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:16.765432+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:16.765432+0000', 'last_peered': '2026-03-31T21:07:16.765432+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:16.765432+0000', 'last_undegraded': '2026-03-31T21:07:16.765432+0000', 'last_fullsized': '2026-03-31T21:07:16.765432+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11582, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15272, 'num_bytes_recovered': 125077514, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:18.586 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:23.586 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:23.586 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:23.587 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:23.587 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:23.696 INFO:teuthology.orchestra.run.vm00.stdout:227633266753 2026-03-31T21:07:23.696 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:23.696 INFO:teuthology.orchestra.run.vm00.stdout:163208757323 2026-03-31T21:07:23.697 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:23.697 INFO:teuthology.orchestra.run.vm00.stdout:206158430274 2026-03-31T21:07:23.698 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:23.709 INFO:teuthology.orchestra.run.vm00.stdout:90194313329 2026-03-31T21:07:23.709 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:23.907 INFO:teuthology.orchestra.run.vm00.stdout:163208757322 2026-03-31T21:07:23.907 INFO:teuthology.orchestra.run.vm00.stdout:227633266752 2026-03-31T21:07:23.912 INFO:teuthology.orchestra.run.vm00.stdout:206158430273 2026-03-31T21:07:23.920 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266753 got 227633266752 for osd.2 2026-03-31T21:07:23.923 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757323 got 163208757322 for osd.0 2026-03-31T21:07:23.930 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430274 got 206158430273 for osd.3 2026-03-31T21:07:23.954 INFO:teuthology.orchestra.run.vm00.stdout:90194313327 2026-03-31T21:07:23.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313329 got 90194313327 for osd.1 2026-03-31T21:07:24.920 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:24.923 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:24.931 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:24.965 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:25.135 INFO:teuthology.orchestra.run.vm00.stdout:227633266753 2026-03-31T21:07:25.149 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266753 got 227633266753 for osd.2 2026-03-31T21:07:25.149 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:25.167 INFO:teuthology.orchestra.run.vm00.stdout:206158430274 2026-03-31T21:07:25.180 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430274 got 206158430274 for osd.3 2026-03-31T21:07:25.180 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:25.181 INFO:teuthology.orchestra.run.vm00.stdout:163208757323 2026-03-31T21:07:25.196 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757323 got 163208757323 for osd.0 2026-03-31T21:07:25.196 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:25.227 INFO:teuthology.orchestra.run.vm00.stdout:90194313329 2026-03-31T21:07:25.238 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313329 got 90194313329 for osd.1 2026-03-31T21:07:25.238 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:25.238 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:25.460 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:25.460 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:25.470 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-03-31T21:07:23.839649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11515,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15341,"num_bytes_recovered":125954570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":311872,"kb_used_data":116360,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282803648,"statfs":{"total":289910292480,"available":289590935552,"internally_reserved":0,"allocated":119152640,"data_stored":118278786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-139,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":1138688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001586"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44243,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:23.585577+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:23.585577+0000","last_peered":"2026-03-31T21:07:23.585577+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:23.585577+0000","last_undegraded":"2026-03-31T21:07:23.585577+0000","last_fullsized":"2026-03-31T21:07:23.585577+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11515,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15339,"num_bytes_recovered":125626378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11515,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15339,"num_bytes_recovered":125626378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171835392,"data_stored":171835392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430274,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757323,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37704,"kb_used_data":8184,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94334136,"statfs":{"total":96636764160,"available":96598155264,"internally_reserved":0,"allocated":8380416,"data_stored":8086742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7589888,"data_stored":7589888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:25.470 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44243, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:23.585577+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:23.585577+0000', 'last_peered': '2026-03-31T21:07:23.585577+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:23.585577+0000', 'last_undegraded': '2026-03-31T21:07:23.585577+0000', 'last_fullsized': '2026-03-31T21:07:23.585577+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11515, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15339, 'num_bytes_recovered': 125626378, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:25.470 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:30.471 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:30.471 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:30.471 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:30.471 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:30.569 INFO:teuthology.orchestra.run.vm00.stdout:163208757326 2026-03-31T21:07:30.569 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:30.575 INFO:teuthology.orchestra.run.vm00.stdout:90194313331 2026-03-31T21:07:30.575 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:30.577 INFO:teuthology.orchestra.run.vm00.stdout:206158430277 2026-03-31T21:07:30.578 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:30.748 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:07:30.745+0000 7f53d2d76640 0 --2- 192.168.123.100:0/90366613 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f53cc0cc4b0 0x7f53cc11c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:07:30.777 INFO:teuthology.orchestra.run.vm00.stdout:90194313330 2026-03-31T21:07:30.779 INFO:teuthology.orchestra.run.vm00.stdout:227633266755 2026-03-31T21:07:30.779 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:30.790 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313331 got 90194313330 for osd.1 2026-03-31T21:07:30.797 INFO:teuthology.orchestra.run.vm00.stdout:163208757325 2026-03-31T21:07:30.809 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757326 got 163208757325 for osd.0 2026-03-31T21:07:30.834 INFO:teuthology.orchestra.run.vm00.stdout:206158430276 2026-03-31T21:07:30.844 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430277 got 206158430276 for osd.3 2026-03-31T21:07:30.974 INFO:teuthology.orchestra.run.vm00.stdout:227633266754 2026-03-31T21:07:30.985 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266755 got 227633266754 for osd.2 2026-03-31T21:07:31.791 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:31.810 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:31.845 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:31.981 INFO:teuthology.orchestra.run.vm00.stdout:90194313330 2026-03-31T21:07:31.985 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:31.998 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313331 got 90194313330 for osd.1 2026-03-31T21:07:32.046 INFO:teuthology.orchestra.run.vm00.stdout:163208757325 2026-03-31T21:07:32.057 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757326 got 163208757325 for osd.0 2026-03-31T21:07:32.084 INFO:teuthology.orchestra.run.vm00.stdout:206158430276 2026-03-31T21:07:32.095 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430277 got 206158430276 for osd.3 2026-03-31T21:07:32.260 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:07:32.258+0000 7fbed9d74640 0 --2- 192.168.123.100:0/356730087 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7fbedc07f480 0x7fbedc1c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:07:32.377 INFO:teuthology.orchestra.run.vm00.stdout:227633266756 2026-03-31T21:07:32.387 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266755 got 227633266756 for osd.2 2026-03-31T21:07:32.387 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:32.998 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:33.058 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:33.096 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:33.200 INFO:teuthology.orchestra.run.vm00.stdout:90194313331 2026-03-31T21:07:33.220 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313331 got 90194313331 for osd.1 2026-03-31T21:07:33.220 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:33.263 INFO:teuthology.orchestra.run.vm00.stdout:163208757326 2026-03-31T21:07:33.274 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757326 got 163208757326 for osd.0 2026-03-31T21:07:33.274 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:33.296 INFO:teuthology.orchestra.run.vm00.stdout:206158430277 2026-03-31T21:07:33.306 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430277 got 206158430277 for osd.3 2026-03-31T21:07:33.306 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:33.306 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:33.532 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:33.532 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:33.542 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2026-03-31T21:07:31.840686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11447,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15409,"num_bytes_recovered":126511626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312148,"kb_used_data":116636,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282803372,"statfs":{"total":289910292480,"available":289590652928,"internally_reserved":0,"allocated":119435264,"data_stored":118561410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-109,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":892928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001597"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44379,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:30.507145+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:30.507145+0000","last_peered":"2026-03-31T21:07:30.507145+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:30.507145+0000","last_undegraded":"2026-03-31T21:07:30.507145+0000","last_fullsized":"2026-03-31T21:07:30.507145+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11447,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15407,"num_bytes_recovered":126183434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":50,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:04:13.996807+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:04:13.996807+0000","last_peered":"2026-03-31T21:04:13.996807+0000","last_clean":"2026-03-31T21:04:13.996807+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:04:13.996807+0000","last_undegraded":"2026-03-31T21:04:13.996807+0000","last_fullsized":"2026-03-31T21:04:13.996807+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11447,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15407,"num_bytes_recovered":126183434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172113920,"data_stored":172113920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430277,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37980,"kb_used_data":8460,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94333860,"statfs":{"total":96636764160,"available":96597872640,"internally_reserved":0,"allocated":8663040,"data_stored":8369366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7868416,"data_stored":7868416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:33.542 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44379, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:30.507145+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:30.507145+0000', 'last_peered': '2026-03-31T21:07:30.507145+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:30.507145+0000', 'last_undegraded': '2026-03-31T21:07:30.507145+0000', 'last_fullsized': '2026-03-31T21:07:30.507145+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11447, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15407, 'num_bytes_recovered': 126183434, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:33.542 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:38.652 INFO:teuthology.orchestra.run.vm00.stdout:163208757328 2026-03-31T21:07:38.652 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:38.653 INFO:teuthology.orchestra.run.vm00.stdout:90194313334 2026-03-31T21:07:38.653 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:38.655 INFO:teuthology.orchestra.run.vm00.stdout:227633266758 2026-03-31T21:07:38.655 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:38.657 INFO:teuthology.orchestra.run.vm00.stdout:206158430279 2026-03-31T21:07:38.657 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:38.863 INFO:teuthology.orchestra.run.vm00.stdout:163208757327 2026-03-31T21:07:38.874 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757328 got 163208757327 for osd.0 2026-03-31T21:07:38.877 INFO:teuthology.orchestra.run.vm00.stdout:90194313333 2026-03-31T21:07:38.879 INFO:teuthology.orchestra.run.vm00.stdout:227633266757 2026-03-31T21:07:38.890 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313334 got 90194313333 for osd.1 2026-03-31T21:07:38.891 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266758 got 227633266757 for osd.2 2026-03-31T21:07:38.924 INFO:teuthology.orchestra.run.vm00.stdout:206158430278 2026-03-31T21:07:38.935 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430279 got 206158430278 for osd.3 2026-03-31T21:07:39.875 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:39.890 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:39.891 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:39.935 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:40.095 INFO:teuthology.orchestra.run.vm00.stdout:163208757327 2026-03-31T21:07:40.111 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757328 got 163208757327 for osd.0 2026-03-31T21:07:40.144 INFO:teuthology.orchestra.run.vm00.stdout:90194313333 2026-03-31T21:07:40.160 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313334 got 90194313333 for osd.1 2026-03-31T21:07:40.173 INFO:teuthology.orchestra.run.vm00.stdout:227633266758 2026-03-31T21:07:40.186 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266758 got 227633266758 for osd.2 2026-03-31T21:07:40.186 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:40.197 INFO:teuthology.orchestra.run.vm00.stdout:206158430280 2026-03-31T21:07:40.208 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430279 got 206158430280 for osd.3 2026-03-31T21:07:40.208 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:41.112 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:41.161 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:41.318 INFO:teuthology.orchestra.run.vm00.stdout:163208757329 2026-03-31T21:07:41.333 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757328 got 163208757329 for osd.0 2026-03-31T21:07:41.333 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:41.369 INFO:teuthology.orchestra.run.vm00.stdout:90194313334 2026-03-31T21:07:41.381 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313334 got 90194313334 for osd.1 2026-03-31T21:07:41.381 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:41.381 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:41.617 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:41.617 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:41.629 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2026-03-31T21:07:39.841794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11360,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15496,"num_bytes_recovered":127224330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312500,"kb_used_data":116988,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282803020,"statfs":{"total":289910292480,"available":289590292480,"internally_reserved":0,"allocated":119795712,"data_stored":118921858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001686"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44553,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:39.366638+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:39.366638+0000","last_peered":"2026-03-31T21:07:39.366638+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:39.366638+0000","last_undegraded":"2026-03-31T21:07:39.366638+0000","last_fullsized":"2026-03-31T21:07:39.366638+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11360,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15494,"num_bytes_recovered":126896138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11360,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15494,"num_bytes_recovered":126896138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172474368,"data_stored":172474368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430280,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757329,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38332,"kb_used_data":8812,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94333508,"statfs":{"total":96636764160,"available":96597512192,"internally_reserved":0,"allocated":9023488,"data_stored":8729814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8228864,"data_stored":8228864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:41.629 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44553, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:39.366638+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:39.366638+0000', 'last_peered': '2026-03-31T21:07:39.366638+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:39.366638+0000', 'last_undegraded': '2026-03-31T21:07:39.366638+0000', 'last_fullsized': '2026-03-31T21:07:39.366638+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11360, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15494, 'num_bytes_recovered': 126896138, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:41.629 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:46.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:46.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:46.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:46.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:46.742 INFO:teuthology.orchestra.run.vm00.stdout:206158430282 2026-03-31T21:07:46.742 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:46.748 INFO:teuthology.orchestra.run.vm00.stdout:227633266761 2026-03-31T21:07:46.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:46.748 INFO:teuthology.orchestra.run.vm00.stdout:163208757331 2026-03-31T21:07:46.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:46.750 INFO:teuthology.orchestra.run.vm00.stdout:90194313336 2026-03-31T21:07:46.750 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:46.959 INFO:teuthology.orchestra.run.vm00.stdout:206158430281 2026-03-31T21:07:46.975 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430282 got 206158430281 for osd.3 2026-03-31T21:07:47.005 INFO:teuthology.orchestra.run.vm00.stdout:227633266759 2026-03-31T21:07:47.019 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266761 got 227633266759 for osd.2 2026-03-31T21:07:47.028 INFO:teuthology.orchestra.run.vm00.stdout:163208757330 2026-03-31T21:07:47.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757331 got 163208757330 for osd.0 2026-03-31T21:07:47.055 INFO:teuthology.orchestra.run.vm00.stdout:90194313335 2026-03-31T21:07:47.067 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313336 got 90194313335 for osd.1 2026-03-31T21:07:47.976 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:48.019 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:48.043 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:48.068 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:48.217 INFO:teuthology.orchestra.run.vm00.stdout:227633266761 2026-03-31T21:07:48.228 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266761 got 227633266761 for osd.2 2026-03-31T21:07:48.229 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:48.249 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:07:48.246+0000 7fb049e90640 0 --2- 192.168.123.100:0/1349428209 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7fb04407eb70 0x7fb04407ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:07:48.265 INFO:teuthology.orchestra.run.vm00.stdout:163208757331 2026-03-31T21:07:48.276 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757331 got 163208757331 for osd.0 2026-03-31T21:07:48.277 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:48.311 INFO:teuthology.orchestra.run.vm00.stdout:90194313337 2026-03-31T21:07:48.327 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313336 got 90194313337 for osd.1 2026-03-31T21:07:48.327 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:48.378 INFO:teuthology.orchestra.run.vm00.stdout:206158430282 2026-03-31T21:07:48.389 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430282 got 206158430282 for osd.3 2026-03-31T21:07:48.390 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:48.390 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:48.578 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:48.579 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:48.589 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-03-31T21:07:47.842910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11288,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15568,"num_bytes_recovered":127814154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":312780,"kb_used_data":117268,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282802740,"statfs":{"total":289910292480,"available":289590005760,"internally_reserved":0,"allocated":120082432,"data_stored":119208578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-121,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001677"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44697,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:46.714127+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:46.714127+0000","last_peered":"2026-03-31T21:07:46.714127+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:46.714127+0000","last_undegraded":"2026-03-31T21:07:46.714127+0000","last_fullsized":"2026-03-31T21:07:46.714127+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15566,"num_bytes_recovered":127485962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15566,"num_bytes_recovered":127485962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":172765184,"data_stored":172765184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430282,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313337,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757331,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38612,"kb_used_data":9092,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94333228,"statfs":{"total":96636764160,"available":96597225472,"internally_reserved":0,"allocated":9310208,"data_stored":9016534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8519680,"data_stored":8519680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:48.590 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44697, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:46.714127+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:46.714127+0000', 'last_peered': '2026-03-31T21:07:46.714127+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:46.714127+0000', 'last_undegraded': '2026-03-31T21:07:46.714127+0000', 'last_fullsized': '2026-03-31T21:07:46.714127+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11288, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15566, 'num_bytes_recovered': 127485962, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:48.590 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:07:53.590 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:07:53.590 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:07:53.590 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:07:53.591 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:07:53.689 INFO:teuthology.orchestra.run.vm00.stdout:227633266763 2026-03-31T21:07:53.689 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:53.692 INFO:teuthology.orchestra.run.vm00.stdout:206158430284 2026-03-31T21:07:53.692 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:53.696 INFO:teuthology.orchestra.run.vm00.stdout:90194313339 2026-03-31T21:07:53.696 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:53.864 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:07:53.861+0000 7f34eeffd640 0 --2- 192.168.123.100:0/368265819 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f34f0131970 0x7f34f0131d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:07:53.895 INFO:teuthology.orchestra.run.vm00.stdout:90194313337 2026-03-31T21:07:53.895 INFO:teuthology.orchestra.run.vm00.stdout:227633266762 2026-03-31T21:07:53.896 INFO:teuthology.orchestra.run.vm00.stdout:163208757333 2026-03-31T21:07:53.897 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:53.903 INFO:teuthology.orchestra.run.vm00.stdout:206158430283 2026-03-31T21:07:53.910 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313339 got 90194313337 for osd.1 2026-03-31T21:07:53.912 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266763 got 227633266762 for osd.2 2026-03-31T21:07:53.916 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430284 got 206158430283 for osd.3 2026-03-31T21:07:54.086 INFO:teuthology.orchestra.run.vm00.stdout:163208757332 2026-03-31T21:07:54.095 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757333 got 163208757332 for osd.0 2026-03-31T21:07:54.911 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:07:54.912 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:07:54.917 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:07:55.096 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:55.106 INFO:teuthology.orchestra.run.vm00.stdout:90194313339 2026-03-31T21:07:55.117 INFO:teuthology.orchestra.run.vm00.stdout:227633266763 2026-03-31T21:07:55.128 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313339 got 90194313339 for osd.1 2026-03-31T21:07:55.128 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:55.131 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266763 got 227633266763 for osd.2 2026-03-31T21:07:55.131 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:55.155 INFO:teuthology.orchestra.run.vm00.stdout:206158430284 2026-03-31T21:07:55.168 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430284 got 206158430284 for osd.3 2026-03-31T21:07:55.169 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:55.294 INFO:teuthology.orchestra.run.vm00.stdout:163208757332 2026-03-31T21:07:55.304 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757333 got 163208757332 for osd.0 2026-03-31T21:07:56.304 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:07:56.487 INFO:teuthology.orchestra.run.vm00.stdout:163208757334 2026-03-31T21:07:56.497 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757333 got 163208757334 for osd.0 2026-03-31T21:07:56.497 DEBUG:teuthology.parallel:result is None 2026-03-31T21:07:56.497 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:07:56.679 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:07:56.679 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:07:56.689 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2026-03-31T21:07:55.843851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11213,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15643,"num_bytes_recovered":128421040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":313092,"kb_used_data":117580,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282802428,"statfs":{"total":289910292480,"available":289589686272,"internally_reserved":0,"allocated":120401920,"data_stored":119528066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1196710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001447"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44847,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:54.362785+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:54.362785+0000","last_peered":"2026-03-31T21:07:54.362785+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:54.362785+0000","last_undegraded":"2026-03-31T21:07:54.362785+0000","last_fullsized":"2026-03-31T21:07:54.362785+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11213,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15641,"num_bytes_recovered":128092848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11213,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15641,"num_bytes_recovered":128092848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173076480,"data_stored":173076480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430285,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266763,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757334,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38924,"kb_used_data":9404,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94332916,"statfs":{"total":96636764160,"available":96596905984,"internally_reserved":0,"allocated":9629696,"data_stored":9336022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8830976,"data_stored":8830976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:07:56.689 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44847, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:54.362785+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:54.362785+0000', 'last_peered': '2026-03-31T21:07:54.362785+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:54.362785+0000', 'last_undegraded': '2026-03-31T21:07:54.362785+0000', 'last_fullsized': '2026-03-31T21:07:54.362785+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11213, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15641, 'num_bytes_recovered': 128092848, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:07:56.689 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:01.690 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:01.691 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:01.691 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:01.691 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:01.789 INFO:teuthology.orchestra.run.vm00.stdout:163208757336 2026-03-31T21:08:01.789 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:01.789 INFO:teuthology.orchestra.run.vm00.stdout:90194313341 2026-03-31T21:08:01.790 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:01.792 INFO:teuthology.orchestra.run.vm00.stdout:206158430287 2026-03-31T21:08:01.793 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:01.801 INFO:teuthology.orchestra.run.vm00.stdout:227633266766 2026-03-31T21:08:01.801 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:01.994 INFO:teuthology.orchestra.run.vm00.stdout:206158430286 2026-03-31T21:08:01.996 INFO:teuthology.orchestra.run.vm00.stdout:90194313340 2026-03-31T21:08:01.998 INFO:teuthology.orchestra.run.vm00.stdout:163208757335 2026-03-31T21:08:02.008 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430287 got 206158430286 for osd.3 2026-03-31T21:08:02.013 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757336 got 163208757335 for osd.0 2026-03-31T21:08:02.013 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313341 got 90194313340 for osd.1 2026-03-31T21:08:02.058 INFO:teuthology.orchestra.run.vm00.stdout:227633266764 2026-03-31T21:08:02.069 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266766 got 227633266764 for osd.2 2026-03-31T21:08:03.009 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:03.013 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:03.014 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:03.069 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:03.208 INFO:teuthology.orchestra.run.vm00.stdout:206158430287 2026-03-31T21:08:03.221 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430287 got 206158430287 for osd.3 2026-03-31T21:08:03.221 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:03.258 INFO:teuthology.orchestra.run.vm00.stdout:163208757336 2026-03-31T21:08:03.274 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757336 got 163208757336 for osd.0 2026-03-31T21:08:03.274 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:03.292 INFO:teuthology.orchestra.run.vm00.stdout:227633266766 2026-03-31T21:08:03.303 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266766 got 227633266766 for osd.2 2026-03-31T21:08:03.303 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:03.323 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:08:03.321+0000 7f9bf178a640 0 --2- 192.168.123.100:0/1079211605 >> v2:192.168.123.100:3300/0 conn(0x7f9bec07f480 0x7f9bec171fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:08:03.441 INFO:teuthology.orchestra.run.vm00.stdout:90194313341 2026-03-31T21:08:03.451 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313341 got 90194313341 for osd.1 2026-03-31T21:08:03.451 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:03.452 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:03.673 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:03.673 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:03.682 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2026-03-31T21:08:01.844505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11169,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15686,"num_bytes_recovered":128773296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":313268,"kb_used_data":117756,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282802252,"statfs":{"total":289910292480,"available":289589506048,"internally_reserved":0,"allocated":120582144,"data_stored":119708290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-93,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001352"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44934,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:58.739345+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:58.739345+0000","last_peered":"2026-03-31T21:07:58.739345+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:58.739345+0000","last_undegraded":"2026-03-31T21:07:58.739345+0000","last_fullsized":"2026-03-31T21:07:58.739345+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11169,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15684,"num_bytes_recovered":128445104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11169,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15684,"num_bytes_recovered":128445104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173248512,"data_stored":173248512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430287,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757336,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39100,"kb_used_data":9580,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94332740,"statfs":{"total":96636764160,"available":96596725760,"internally_reserved":0,"allocated":9809920,"data_stored":9516246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9003008,"data_stored":9003008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:03.682 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44934, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:58.739345+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:58.739345+0000', 'last_peered': '2026-03-31T21:07:58.739345+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:58.739345+0000', 'last_undegraded': '2026-03-31T21:07:58.739345+0000', 'last_fullsized': '2026-03-31T21:07:58.739345+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11169, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15684, 'num_bytes_recovered': 128445104, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:03.682 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:08.683 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:08.684 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:08.684 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:08.684 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:08.785 INFO:teuthology.orchestra.run.vm00.stdout:163208757338 2026-03-31T21:08:08.785 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:08.786 INFO:teuthology.orchestra.run.vm00.stdout:90194313344 2026-03-31T21:08:08.786 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:08.786 INFO:teuthology.orchestra.run.vm00.stdout:206158430289 2026-03-31T21:08:08.786 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:08.790 INFO:teuthology.orchestra.run.vm00.stdout:227633266768 2026-03-31T21:08:08.790 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:08.990 INFO:teuthology.orchestra.run.vm00.stdout:163208757337 2026-03-31T21:08:08.991 INFO:teuthology.orchestra.run.vm00.stdout:206158430288 2026-03-31T21:08:08.992 INFO:teuthology.orchestra.run.vm00.stdout:90194313343 2026-03-31T21:08:08.999 INFO:teuthology.orchestra.run.vm00.stdout:227633266767 2026-03-31T21:08:09.005 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757338 got 163208757337 for osd.0 2026-03-31T21:08:09.005 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430289 got 206158430288 for osd.3 2026-03-31T21:08:09.008 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313344 got 90194313343 for osd.1 2026-03-31T21:08:09.011 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266768 got 227633266767 for osd.2 2026-03-31T21:08:10.005 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:10.006 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:10.008 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:10.011 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:10.222 INFO:teuthology.orchestra.run.vm00.stdout:206158430290 2026-03-31T21:08:10.238 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430289 got 206158430290 for osd.3 2026-03-31T21:08:10.238 INFO:teuthology.orchestra.run.vm00.stdout:163208757339 2026-03-31T21:08:10.238 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:10.249 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757338 got 163208757339 for osd.0 2026-03-31T21:08:10.249 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:10.260 INFO:teuthology.orchestra.run.vm00.stdout:227633266768 2026-03-31T21:08:10.270 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266768 got 227633266768 for osd.2 2026-03-31T21:08:10.270 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:10.279 INFO:teuthology.orchestra.run.vm00.stdout:90194313344 2026-03-31T21:08:10.291 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313344 got 90194313344 for osd.1 2026-03-31T21:08:10.291 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:10.291 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:10.514 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:10.514 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:10.524 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2026-03-31T21:08:09.845617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11169,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15686,"num_bytes_recovered":128773296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":313268,"kb_used_data":117756,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282802252,"statfs":{"total":289910292480,"available":289589506048,"internally_reserved":0,"allocated":120582144,"data_stored":119708290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-44,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":352256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44934,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:07:58.739345+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:07:58.739345+0000","last_peered":"2026-03-31T21:07:58.739345+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:07:58.739345+0000","last_undegraded":"2026-03-31T21:07:58.739345+0000","last_fullsized":"2026-03-31T21:07:58.739345+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11169,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15684,"num_bytes_recovered":128445104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11169,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15684,"num_bytes_recovered":128445104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173248512,"data_stored":173248512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313344,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757339,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39100,"kb_used_data":9580,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94332740,"statfs":{"total":96636764160,"available":96596725760,"internally_reserved":0,"allocated":9809920,"data_stored":9516246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9003008,"data_stored":9003008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:10.524 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44934, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:07:58.739345+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:07:58.739345+0000', 'last_peered': '2026-03-31T21:07:58.739345+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:07:58.739345+0000', 'last_undegraded': '2026-03-31T21:07:58.739345+0000', 'last_fullsized': '2026-03-31T21:07:58.739345+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11169, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15684, 'num_bytes_recovered': 128445104, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:10.525 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:15.525 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:15.525 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:15.526 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:15.526 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:15.642 INFO:teuthology.orchestra.run.vm00.stdout:163208757341 2026-03-31T21:08:15.642 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:15.660 INFO:teuthology.orchestra.run.vm00.stdout:90194313346 2026-03-31T21:08:15.661 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:15.667 INFO:teuthology.orchestra.run.vm00.stdout:227633266770 2026-03-31T21:08:15.667 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:15.671 INFO:teuthology.orchestra.run.vm00.stdout:206158430292 2026-03-31T21:08:15.671 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:15.886 INFO:teuthology.orchestra.run.vm00.stdout:163208757339 2026-03-31T21:08:15.900 INFO:teuthology.orchestra.run.vm00.stdout:206158430290 2026-03-31T21:08:15.904 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757341 got 163208757339 for osd.0 2026-03-31T21:08:15.909 INFO:teuthology.orchestra.run.vm00.stdout:227633266769 2026-03-31T21:08:15.911 INFO:teuthology.orchestra.run.vm00.stdout:90194313345 2026-03-31T21:08:15.917 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430292 got 206158430290 for osd.3 2026-03-31T21:08:15.925 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266770 got 227633266769 for osd.2 2026-03-31T21:08:15.927 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313346 got 90194313345 for osd.1 2026-03-31T21:08:16.905 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:16.918 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:16.926 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:16.927 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:17.138 INFO:teuthology.orchestra.run.vm00.stdout:163208757341 2026-03-31T21:08:17.158 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757341 got 163208757341 for osd.0 2026-03-31T21:08:17.158 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:17.171 INFO:teuthology.orchestra.run.vm00.stdout:90194313346 2026-03-31T21:08:17.183 INFO:teuthology.orchestra.run.vm00.stdout:206158430292 2026-03-31T21:08:17.187 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313346 got 90194313346 for osd.1 2026-03-31T21:08:17.187 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:17.188 INFO:teuthology.orchestra.run.vm00.stdout:227633266770 2026-03-31T21:08:17.198 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430292 got 206158430292 for osd.3 2026-03-31T21:08:17.198 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:17.200 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266770 got 227633266770 for osd.2 2026-03-31T21:08:17.200 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:17.200 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:17.439 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:17.439 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:17.451 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2026-03-31T21:08:15.846395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11152,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15704,"num_bytes_recovered":128920752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":313328,"kb_used_data":117816,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282802192,"statfs":{"total":289910292480,"available":289589444608,"internally_reserved":0,"allocated":120643584,"data_stored":119769730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-17,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":147456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":44970,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:08:15.614849+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:08:15.614849+0000","last_peered":"2026-03-31T21:08:15.614849+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:08:15.614849+0000","last_undegraded":"2026-03-31T21:08:15.614849+0000","last_fullsized":"2026-03-31T21:08:15.614849+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11152,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15702,"num_bytes_recovered":128592560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11152,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15702,"num_bytes_recovered":128592560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173322240,"data_stored":173322240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430292,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757341,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39160,"kb_used_data":9640,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94332680,"statfs":{"total":96636764160,"available":96596664320,"internally_reserved":0,"allocated":9871360,"data_stored":9577686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9076736,"data_stored":9076736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:17.452 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 44970, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:08:15.614849+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:08:15.614849+0000', 'last_peered': '2026-03-31T21:08:15.614849+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:08:15.614849+0000', 'last_undegraded': '2026-03-31T21:08:15.614849+0000', 'last_fullsized': '2026-03-31T21:08:15.614849+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11152, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15702, 'num_bytes_recovered': 128592560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:17.452 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:22.452 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:22.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:22.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:22.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:22.564 INFO:teuthology.orchestra.run.vm00.stdout:206158430294 2026-03-31T21:08:22.564 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:22.567 INFO:teuthology.orchestra.run.vm00.stdout:90194313348 2026-03-31T21:08:22.567 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:22.578 INFO:teuthology.orchestra.run.vm00.stdout:227633266773 2026-03-31T21:08:22.578 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:22.580 INFO:teuthology.orchestra.run.vm00.stdout:163208757343 2026-03-31T21:08:22.580 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:22.796 INFO:teuthology.orchestra.run.vm00.stdout:163208757342 2026-03-31T21:08:22.797 INFO:teuthology.orchestra.run.vm00.stdout:206158430293 2026-03-31T21:08:22.807 INFO:teuthology.orchestra.run.vm00.stdout:90194313347 2026-03-31T21:08:22.811 INFO:teuthology.orchestra.run.vm00.stdout:227633266772 2026-03-31T21:08:22.814 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430294 got 206158430293 for osd.3 2026-03-31T21:08:22.821 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757343 got 163208757342 for osd.0 2026-03-31T21:08:22.821 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313348 got 90194313347 for osd.1 2026-03-31T21:08:22.828 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266773 got 227633266772 for osd.2 2026-03-31T21:08:23.815 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:23.821 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:23.822 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:23.829 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:24.042 INFO:teuthology.orchestra.run.vm00.stdout:206158430294 2026-03-31T21:08:24.060 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430294 got 206158430294 for osd.3 2026-03-31T21:08:24.060 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:24.079 INFO:teuthology.orchestra.run.vm00.stdout:227633266773 2026-03-31T21:08:24.095 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266773 got 227633266773 for osd.2 2026-03-31T21:08:24.095 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:24.108 INFO:teuthology.orchestra.run.vm00.stdout:163208757343 2026-03-31T21:08:24.108 INFO:teuthology.orchestra.run.vm00.stdout:90194313349 2026-03-31T21:08:24.122 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313348 got 90194313349 for osd.1 2026-03-31T21:08:24.122 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:24.122 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757343 got 163208757343 for osd.0 2026-03-31T21:08:24.122 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:24.122 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:24.362 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:24.362 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:24.374 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2026-03-31T21:08:23.847543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11084,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15772,"num_bytes_recovered":129477808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":313600,"kb_used_data":118088,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282801920,"statfs":{"total":289910292480,"available":289589166080,"internally_reserved":0,"allocated":120922112,"data_stored":120048258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-85,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001671"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":45106,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:08:22.553535+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:08:22.553535+0000","last_peered":"2026-03-31T21:08:22.553535+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:08:22.553535+0000","last_undegraded":"2026-03-31T21:08:22.553535+0000","last_fullsized":"2026-03-31T21:08:22.553535+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11084,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15770,"num_bytes_recovered":129149616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11084,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15770,"num_bytes_recovered":129149616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173600768,"data_stored":173600768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266773,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313349,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757343,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39432,"kb_used_data":9912,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94332408,"statfs":{"total":96636764160,"available":96596385792,"internally_reserved":0,"allocated":10149888,"data_stored":9856214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9355264,"data_stored":9355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:24.374 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 45106, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:08:22.553535+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:08:22.553535+0000', 'last_peered': '2026-03-31T21:08:22.553535+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:08:22.553535+0000', 'last_undegraded': '2026-03-31T21:08:22.553535+0000', 'last_fullsized': '2026-03-31T21:08:22.553535+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11084, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15770, 'num_bytes_recovered': 129149616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:24.374 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:29.375 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:29.375 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:29.376 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:29.376 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:29.480 INFO:teuthology.orchestra.run.vm00.stdout:90194313351 2026-03-31T21:08:29.480 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:29.486 INFO:teuthology.orchestra.run.vm00.stdout:163208757345 2026-03-31T21:08:29.487 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:29.487 INFO:teuthology.orchestra.run.vm00.stdout:206158430297 2026-03-31T21:08:29.488 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:29.503 INFO:teuthology.orchestra.run.vm00.stdout:227633266775 2026-03-31T21:08:29.503 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:29.700 INFO:teuthology.orchestra.run.vm00.stdout:90194313350 2026-03-31T21:08:29.705 INFO:teuthology.orchestra.run.vm00.stdout:163208757344 2026-03-31T21:08:29.705 INFO:teuthology.orchestra.run.vm00.stdout:206158430295 2026-03-31T21:08:29.717 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430297 got 206158430295 for osd.3 2026-03-31T21:08:29.718 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313351 got 90194313350 for osd.1 2026-03-31T21:08:29.718 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757345 got 163208757344 for osd.0 2026-03-31T21:08:29.754 INFO:teuthology.orchestra.run.vm00.stdout:227633266774 2026-03-31T21:08:29.765 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266775 got 227633266774 for osd.2 2026-03-31T21:08:30.718 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:30.719 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:30.719 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:30.765 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:30.920 INFO:teuthology.orchestra.run.vm00.stdout:163208757346 2026-03-31T21:08:30.927 INFO:teuthology.orchestra.run.vm00.stdout:206158430297 2026-03-31T21:08:30.933 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757345 got 163208757346 for osd.0 2026-03-31T21:08:30.933 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:30.941 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430297 got 206158430297 for osd.3 2026-03-31T21:08:30.941 INFO:teuthology.orchestra.run.vm00.stdout:90194313351 2026-03-31T21:08:30.941 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:30.954 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313351 got 90194313351 for osd.1 2026-03-31T21:08:30.954 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:30.992 INFO:teuthology.orchestra.run.vm00.stdout:227633266775 2026-03-31T21:08:31.003 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266775 got 227633266775 for osd.2 2026-03-31T21:08:31.003 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:31.003 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:31.228 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:31.228 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:31.237 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2026-03-31T21:08:29.848304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11017,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15839,"num_bytes_recovered":130026672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":313872,"kb_used_data":118360,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282801648,"statfs":{"total":289910292480,"available":289588887552,"internally_reserved":0,"allocated":121200640,"data_stored":120326786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-135,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001626"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":45240,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:08:29.392612+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:08:29.392612+0000","last_peered":"2026-03-31T21:08:29.392612+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:08:29.392612+0000","last_undegraded":"2026-03-31T21:08:29.392612+0000","last_fullsized":"2026-03-31T21:08:29.392612+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11017,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15837,"num_bytes_recovered":129698480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11017,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15837,"num_bytes_recovered":129698480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":173879296,"data_stored":173879296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266775,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757346,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39704,"kb_used_data":10184,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94332136,"statfs":{"total":96636764160,"available":96596107264,"internally_reserved":0,"allocated":10428416,"data_stored":10134742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9633792,"data_stored":9633792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:31.238 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 45240, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:08:29.392612+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:08:29.392612+0000', 'last_peered': '2026-03-31T21:08:29.392612+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:08:29.392612+0000', 'last_undegraded': '2026-03-31T21:08:29.392612+0000', 'last_fullsized': '2026-03-31T21:08:29.392612+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 11017, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15837, 'num_bytes_recovered': 129698480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:31.238 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:36.238 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:36.239 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:36.239 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:36.239 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:36.341 INFO:teuthology.orchestra.run.vm00.stdout:90194313353 2026-03-31T21:08:36.341 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:36.353 INFO:teuthology.orchestra.run.vm00.stdout:206158430299 2026-03-31T21:08:36.353 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:36.355 INFO:teuthology.orchestra.run.vm00.stdout:163208757348 2026-03-31T21:08:36.355 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:36.359 INFO:teuthology.orchestra.run.vm00.stdout:227633266777 2026-03-31T21:08:36.359 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:36.565 INFO:teuthology.orchestra.run.vm00.stdout:90194313352 2026-03-31T21:08:36.565 INFO:teuthology.orchestra.run.vm00.stdout:206158430298 2026-03-31T21:08:36.565 INFO:teuthology.orchestra.run.vm00.stdout:163208757347 2026-03-31T21:08:36.569 INFO:teuthology.orchestra.run.vm00.stdout:227633266776 2026-03-31T21:08:36.578 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313353 got 90194313352 for osd.1 2026-03-31T21:08:36.579 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430299 got 206158430298 for osd.3 2026-03-31T21:08:36.582 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757348 got 163208757347 for osd.0 2026-03-31T21:08:36.585 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266777 got 227633266776 for osd.2 2026-03-31T21:08:37.579 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:37.579 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:37.583 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:37.586 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:37.787 INFO:teuthology.orchestra.run.vm00.stdout:206158430298 2026-03-31T21:08:37.796 INFO:teuthology.orchestra.run.vm00.stdout:90194313352 2026-03-31T21:08:37.802 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430299 got 206158430298 for osd.3 2026-03-31T21:08:37.810 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313353 got 90194313352 for osd.1 2026-03-31T21:08:37.830 INFO:teuthology.orchestra.run.vm00.stdout:163208757347 2026-03-31T21:08:37.841 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757348 got 163208757347 for osd.0 2026-03-31T21:08:37.850 INFO:teuthology.orchestra.run.vm00.stdout:227633266776 2026-03-31T21:08:37.861 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266777 got 227633266776 for osd.2 2026-03-31T21:08:38.803 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:38.811 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:38.841 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:38.862 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:39.010 INFO:teuthology.orchestra.run.vm00.stdout:206158430299 2026-03-31T21:08:39.022 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430299 got 206158430299 for osd.3 2026-03-31T21:08:39.022 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:39.048 INFO:teuthology.orchestra.run.vm00.stdout:90194313354 2026-03-31T21:08:39.060 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313353 got 90194313354 for osd.1 2026-03-31T21:08:39.060 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:39.074 INFO:teuthology.orchestra.run.vm00.stdout:163208757348 2026-03-31T21:08:39.086 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757348 got 163208757348 for osd.0 2026-03-31T21:08:39.086 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:39.094 INFO:teuthology.orchestra.run.vm00.stdout:227633266778 2026-03-31T21:08:39.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266777 got 227633266778 for osd.2 2026-03-31T21:08:39.104 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:39.104 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:39.324 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:39.325 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:39.334 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2026-03-31T21:08:37.849411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10950,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15906,"num_bytes_recovered":130575536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314132,"kb_used_data":118620,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282801388,"statfs":{"total":289910292480,"available":289588621312,"internally_reserved":0,"allocated":121466880,"data_stored":120593026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-116,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":45374,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:08:36.312902+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:08:36.312902+0000","last_peered":"2026-03-31T21:08:36.312902+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:08:36.312902+0000","last_undegraded":"2026-03-31T21:08:36.312902+0000","last_fullsized":"2026-03-31T21:08:36.312902+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10950,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15904,"num_bytes_recovered":130247344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10950,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15904,"num_bytes_recovered":130247344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174149632,"data_stored":174149632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266778,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313354,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757348,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39964,"kb_used_data":10444,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94331876,"statfs":{"total":96636764160,"available":96595841024,"internally_reserved":0,"allocated":10694656,"data_stored":10400982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":9904128,"data_stored":9904128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:39.335 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 45374, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:08:36.312902+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:08:36.312902+0000', 'last_peered': '2026-03-31T21:08:36.312902+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:08:36.312902+0000', 'last_undegraded': '2026-03-31T21:08:36.312902+0000', 'last_fullsized': '2026-03-31T21:08:36.312902+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10950, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15904, 'num_bytes_recovered': 130247344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:39.335 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:44.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:44.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:44.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:44.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:44.444 INFO:teuthology.orchestra.run.vm00.stdout:206158430302 2026-03-31T21:08:44.444 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:44.449 INFO:teuthology.orchestra.run.vm00.stdout:227633266780 2026-03-31T21:08:44.450 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:44.451 INFO:teuthology.orchestra.run.vm00.stdout:163208757350 2026-03-31T21:08:44.451 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:44.451 INFO:teuthology.orchestra.run.vm00.stdout:90194313356 2026-03-31T21:08:44.451 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:44.641 INFO:teuthology.orchestra.run.vm00.stdout:206158430300 2026-03-31T21:08:44.657 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430302 got 206158430300 for osd.3 2026-03-31T21:08:44.696 INFO:teuthology.orchestra.run.vm00.stdout:227633266779 2026-03-31T21:08:44.705 INFO:teuthology.orchestra.run.vm00.stdout:163208757349 2026-03-31T21:08:44.711 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266780 got 227633266779 for osd.2 2026-03-31T21:08:44.718 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757350 got 163208757349 for osd.0 2026-03-31T21:08:44.732 INFO:teuthology.orchestra.run.vm00.stdout:90194313355 2026-03-31T21:08:44.743 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313356 got 90194313355 for osd.1 2026-03-31T21:08:45.658 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:45.712 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:45.718 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:45.744 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:45.857 INFO:teuthology.orchestra.run.vm00.stdout:206158430300 2026-03-31T21:08:45.873 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430302 got 206158430300 for osd.3 2026-03-31T21:08:45.922 INFO:teuthology.orchestra.run.vm00.stdout:227633266779 2026-03-31T21:08:45.938 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266780 got 227633266779 for osd.2 2026-03-31T21:08:45.962 INFO:teuthology.orchestra.run.vm00.stdout:163208757349 2026-03-31T21:08:45.965 INFO:teuthology.orchestra.run.vm00.stdout:90194313355 2026-03-31T21:08:45.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757350 got 163208757349 for osd.0 2026-03-31T21:08:45.976 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313356 got 90194313355 for osd.1 2026-03-31T21:08:46.873 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:46.939 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:46.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:46.977 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:47.078 INFO:teuthology.orchestra.run.vm00.stdout:206158430302 2026-03-31T21:08:47.092 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430302 got 206158430302 for osd.3 2026-03-31T21:08:47.092 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:47.141 INFO:teuthology.orchestra.run.vm00.stdout:227633266780 2026-03-31T21:08:47.159 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266780 got 227633266780 for osd.2 2026-03-31T21:08:47.159 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:47.190 INFO:teuthology.orchestra.run.vm00.stdout:163208757351 2026-03-31T21:08:47.192 INFO:teuthology.orchestra.run.vm00.stdout:90194313356 2026-03-31T21:08:47.202 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757350 got 163208757351 for osd.0 2026-03-31T21:08:47.202 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:47.205 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313356 got 90194313356 for osd.1 2026-03-31T21:08:47.205 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:47.205 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:47.427 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:47.427 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:47.436 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2026-03-31T21:08:45.850297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10871,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15985,"num_bytes_recovered":131222704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314452,"kb_used_data":118940,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282801068,"statfs":{"total":289910292480,"available":289588293632,"internally_reserved":0,"allocated":121794560,"data_stored":120920706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001479"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":45532,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:08:44.346965+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:08:44.346965+0000","last_peered":"2026-03-31T21:08:44.346965+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:08:44.346965+0000","last_undegraded":"2026-03-31T21:08:44.346965+0000","last_fullsized":"2026-03-31T21:08:44.346965+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10871,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15983,"num_bytes_recovered":130894512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10871,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15983,"num_bytes_recovered":130894512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174477312,"data_stored":174477312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430302,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757351,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40284,"kb_used_data":10764,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94331556,"statfs":{"total":96636764160,"available":96595513344,"internally_reserved":0,"allocated":11022336,"data_stored":10728662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10231808,"data_stored":10231808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:47.437 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 45532, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:08:44.346965+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:08:44.346965+0000', 'last_peered': '2026-03-31T21:08:44.346965+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:08:44.346965+0000', 'last_undegraded': '2026-03-31T21:08:44.346965+0000', 'last_fullsized': '2026-03-31T21:08:44.346965+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10871, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 15983, 'num_bytes_recovered': 130894512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:47.437 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:52.437 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:52.437 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:52.437 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:52.437 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:52.537 INFO:teuthology.orchestra.run.vm00.stdout:90194313358 2026-03-31T21:08:52.537 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:52.537 INFO:teuthology.orchestra.run.vm00.stdout:227633266783 2026-03-31T21:08:52.537 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:52.542 INFO:teuthology.orchestra.run.vm00.stdout:206158430304 2026-03-31T21:08:52.542 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:52.712 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:08:52.710+0000 7f9702847640 0 --2- 192.168.123.100:0/1955915264 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f96fc07dfc0 0x7f96fc069030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:08:52.735 INFO:teuthology.orchestra.run.vm00.stdout:163208757353 2026-03-31T21:08:52.735 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:52.740 INFO:teuthology.orchestra.run.vm00.stdout:206158430303 2026-03-31T21:08:52.742 INFO:teuthology.orchestra.run.vm00.stdout:90194313357 2026-03-31T21:08:52.751 INFO:teuthology.orchestra.run.vm00.stdout:227633266782 2026-03-31T21:08:52.753 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430304 got 206158430303 for osd.3 2026-03-31T21:08:52.761 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313358 got 90194313357 for osd.1 2026-03-31T21:08:52.765 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266783 got 227633266782 for osd.2 2026-03-31T21:08:52.933 INFO:teuthology.orchestra.run.vm00.stdout:163208757352 2026-03-31T21:08:52.944 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757353 got 163208757352 for osd.0 2026-03-31T21:08:53.753 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:53.762 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:53.765 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:53.945 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:53.969 INFO:teuthology.orchestra.run.vm00.stdout:206158430304 2026-03-31T21:08:53.986 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430304 got 206158430304 for osd.3 2026-03-31T21:08:53.986 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:54.010 INFO:teuthology.orchestra.run.vm00.stdout:90194313359 2026-03-31T21:08:54.017 INFO:teuthology.orchestra.run.vm00.stdout:227633266783 2026-03-31T21:08:54.031 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266783 got 227633266783 for osd.2 2026-03-31T21:08:54.031 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:54.031 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313358 got 90194313359 for osd.1 2026-03-31T21:08:54.031 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:54.157 INFO:teuthology.orchestra.run.vm00.stdout:163208757353 2026-03-31T21:08:54.169 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757353 got 163208757353 for osd.0 2026-03-31T21:08:54.169 DEBUG:teuthology.parallel:result is None 2026-03-31T21:08:54.169 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:08:54.354 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:08:54.354 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:08:54.364 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2026-03-31T21:08:53.851186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10791,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16065,"num_bytes_recovered":131878064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":314760,"kb_used_data":119248,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282800760,"statfs":{"total":289910292480,"available":289587978240,"internally_reserved":0,"allocated":122109952,"data_stored":121236098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-129,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001291"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":45692,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:08:52.485003+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:08:52.485003+0000","last_peered":"2026-03-31T21:08:52.485003+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:08:52.485003+0000","last_undegraded":"2026-03-31T21:08:52.485003+0000","last_fullsized":"2026-03-31T21:08:52.485003+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10791,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16063,"num_bytes_recovered":131549872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10791,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16063,"num_bytes_recovered":131549872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":174809088,"data_stored":174809088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430304,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757353,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40592,"kb_used_data":11072,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94331248,"statfs":{"total":96636764160,"available":96595197952,"internally_reserved":0,"allocated":11337728,"data_stored":11044054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10563584,"data_stored":10563584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:08:54.365 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 45692, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:08:52.485003+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:08:52.485003+0000', 'last_peered': '2026-03-31T21:08:52.485003+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:08:52.485003+0000', 'last_undegraded': '2026-03-31T21:08:52.485003+0000', 'last_fullsized': '2026-03-31T21:08:52.485003+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10791, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16063, 'num_bytes_recovered': 131549872, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:08:54.365 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:08:59.366 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:08:59.366 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:08:59.367 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:08:59.367 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:08:59.466 INFO:teuthology.orchestra.run.vm00.stdout:163208757355 2026-03-31T21:08:59.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:08:59.467 INFO:teuthology.orchestra.run.vm00.stdout:206158430307 2026-03-31T21:08:59.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:08:59.672 INFO:teuthology.orchestra.run.vm00.stdout:227633266785 2026-03-31T21:08:59.673 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:08:59.673 INFO:teuthology.orchestra.run.vm00.stdout:90194313361 2026-03-31T21:08:59.673 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:08:59.699 INFO:teuthology.orchestra.run.vm00.stdout:163208757354 2026-03-31T21:08:59.701 INFO:teuthology.orchestra.run.vm00.stdout:206158430305 2026-03-31T21:08:59.716 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430307 got 206158430305 for osd.3 2026-03-31T21:08:59.716 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757355 got 163208757354 for osd.0 2026-03-31T21:08:59.893 INFO:teuthology.orchestra.run.vm00.stdout:227633266784 2026-03-31T21:08:59.893 INFO:teuthology.orchestra.run.vm00.stdout:90194313360 2026-03-31T21:08:59.904 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266785 got 227633266784 for osd.2 2026-03-31T21:08:59.906 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313361 got 90194313360 for osd.1 2026-03-31T21:09:00.717 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:00.717 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:00.905 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:00.905 INFO:teuthology.orchestra.run.vm00.stdout:163208757356 2026-03-31T21:09:00.906 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:00.908 INFO:teuthology.orchestra.run.vm00.stdout:206158430307 2026-03-31T21:09:00.919 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757355 got 163208757356 for osd.0 2026-03-31T21:09:00.920 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:00.925 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430307 got 206158430307 for osd.3 2026-03-31T21:09:00.925 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:01.097 INFO:teuthology.orchestra.run.vm00.stdout:227633266785 2026-03-31T21:09:01.103 INFO:teuthology.orchestra.run.vm00.stdout:90194313361 2026-03-31T21:09:01.109 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266785 got 227633266785 for osd.2 2026-03-31T21:09:01.109 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:01.115 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313361 got 90194313361 for osd.1 2026-03-31T21:09:01.115 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:01.115 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:01.337 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:01.337 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:01.347 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2026-03-31T21:08:59.851852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10723,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16133,"num_bytes_recovered":132435120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":315040,"kb_used_data":119528,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282800480,"statfs":{"total":289910292480,"available":289587691520,"internally_reserved":0,"allocated":122396672,"data_stored":121522818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-148,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001316"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":45828,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:08:59.408633+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:08:59.408633+0000","last_peered":"2026-03-31T21:08:59.408633+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:08:59.408633+0000","last_undegraded":"2026-03-31T21:08:59.408633+0000","last_fullsized":"2026-03-31T21:08:59.408633+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10723,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16131,"num_bytes_recovered":132106928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10723,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16131,"num_bytes_recovered":132106928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":175083520,"data_stored":175083520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430307,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757356,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40872,"kb_used_data":11352,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94330968,"statfs":{"total":96636764160,"available":96594911232,"internally_reserved":0,"allocated":11624448,"data_stored":11330774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":10838016,"data_stored":10838016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:01.347 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 45828, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:08:59.408633+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:08:59.408633+0000', 'last_peered': '2026-03-31T21:08:59.408633+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:08:59.408633+0000', 'last_undegraded': '2026-03-31T21:08:59.408633+0000', 'last_fullsized': '2026-03-31T21:08:59.408633+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10723, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16131, 'num_bytes_recovered': 132106928, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:01.348 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:09:06.348 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:09:06.349 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:09:06.349 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:09:06.349 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:09:06.448 INFO:teuthology.orchestra.run.vm00.stdout:227633266787 2026-03-31T21:09:06.449 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:06.451 INFO:teuthology.orchestra.run.vm00.stdout:206158430309 2026-03-31T21:09:06.451 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:06.453 INFO:teuthology.orchestra.run.vm00.stdout:163208757358 2026-03-31T21:09:06.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:06.460 INFO:teuthology.orchestra.run.vm00.stdout:90194313363 2026-03-31T21:09:06.460 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:06.662 INFO:teuthology.orchestra.run.vm00.stdout:163208757357 2026-03-31T21:09:06.664 INFO:teuthology.orchestra.run.vm00.stdout:90194313362 2026-03-31T21:09:06.666 INFO:teuthology.orchestra.run.vm00.stdout:206158430308 2026-03-31T21:09:06.666 INFO:teuthology.orchestra.run.vm00.stdout:227633266786 2026-03-31T21:09:06.677 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757358 got 163208757357 for osd.0 2026-03-31T21:09:06.678 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313363 got 90194313362 for osd.1 2026-03-31T21:09:06.679 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430309 got 206158430308 for osd.3 2026-03-31T21:09:06.685 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266787 got 227633266786 for osd.2 2026-03-31T21:09:07.678 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:07.678 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:07.680 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:07.685 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:07.881 INFO:teuthology.orchestra.run.vm00.stdout:90194313362 2026-03-31T21:09:07.888 INFO:teuthology.orchestra.run.vm00.stdout:163208757357 2026-03-31T21:09:07.892 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313363 got 90194313362 for osd.1 2026-03-31T21:09:07.892 INFO:teuthology.orchestra.run.vm00.stdout:206158430308 2026-03-31T21:09:07.901 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757358 got 163208757357 for osd.0 2026-03-31T21:09:07.906 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430309 got 206158430308 for osd.3 2026-03-31T21:09:07.937 INFO:teuthology.orchestra.run.vm00.stdout:227633266786 2026-03-31T21:09:07.947 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266787 got 227633266786 for osd.2 2026-03-31T21:09:08.893 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:08.901 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:08.907 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:08.947 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:09.107 INFO:teuthology.orchestra.run.vm00.stdout:90194313364 2026-03-31T21:09:09.118 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313363 got 90194313364 for osd.1 2026-03-31T21:09:09.118 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:09.136 INFO:teuthology.orchestra.run.vm00.stdout:206158430309 2026-03-31T21:09:09.140 INFO:teuthology.orchestra.run.vm00.stdout:163208757358 2026-03-31T21:09:09.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430309 got 206158430309 for osd.3 2026-03-31T21:09:09.148 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:09.153 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757358 got 163208757358 for osd.0 2026-03-31T21:09:09.153 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:09.188 INFO:teuthology.orchestra.run.vm00.stdout:227633266788 2026-03-31T21:09:09.199 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266787 got 227633266788 for osd.2 2026-03-31T21:09:09.199 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:09.199 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:09.421 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:09.421 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:09.430 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2026-03-31T21:09:07.852918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10654,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16202,"num_bytes_recovered":133000368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":315316,"kb_used_data":119804,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282800204,"statfs":{"total":289910292480,"available":289587408896,"internally_reserved":0,"allocated":122679296,"data_stored":121805442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-119,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001491"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":45966,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:09:06.427980+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:09:06.427980+0000","last_peered":"2026-03-31T21:09:06.427980+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:09:06.427980+0000","last_undegraded":"2026-03-31T21:09:06.427980+0000","last_fullsized":"2026-03-31T21:09:06.427980+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10654,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16200,"num_bytes_recovered":132672176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10654,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16200,"num_bytes_recovered":132672176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":175362048,"data_stored":175362048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430309,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313364,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41148,"kb_used_data":11628,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94330692,"statfs":{"total":96636764160,"available":96594628608,"internally_reserved":0,"allocated":11907072,"data_stored":11613398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11116544,"data_stored":11116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:09.431 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 45966, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:09:06.427980+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:09:06.427980+0000', 'last_peered': '2026-03-31T21:09:06.427980+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:09:06.427980+0000', 'last_undegraded': '2026-03-31T21:09:06.427980+0000', 'last_fullsized': '2026-03-31T21:09:06.427980+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10654, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16200, 'num_bytes_recovered': 132672176, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:09.431 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:09:14.432 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:09:14.432 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:09:14.432 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:09:14.432 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:09:14.537 INFO:teuthology.orchestra.run.vm00.stdout:227633266790 2026-03-31T21:09:14.537 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:14.539 INFO:teuthology.orchestra.run.vm00.stdout:163208757360 2026-03-31T21:09:14.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:14.545 INFO:teuthology.orchestra.run.vm00.stdout:206158430312 2026-03-31T21:09:14.545 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:14.546 INFO:teuthology.orchestra.run.vm00.stdout:90194313366 2026-03-31T21:09:14.546 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:14.745 INFO:teuthology.orchestra.run.vm00.stdout:227633266789 2026-03-31T21:09:14.748 INFO:teuthology.orchestra.run.vm00.stdout:206158430310 2026-03-31T21:09:14.752 INFO:teuthology.orchestra.run.vm00.stdout:163208757359 2026-03-31T21:09:14.758 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266790 got 227633266789 for osd.2 2026-03-31T21:09:14.763 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430312 got 206158430310 for osd.3 2026-03-31T21:09:14.765 INFO:teuthology.orchestra.run.vm00.stdout:90194313365 2026-03-31T21:09:14.768 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757360 got 163208757359 for osd.0 2026-03-31T21:09:14.777 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313366 got 90194313365 for osd.1 2026-03-31T21:09:15.759 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:15.764 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:15.769 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:15.778 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:15.974 INFO:teuthology.orchestra.run.vm00.stdout:227633266789 2026-03-31T21:09:15.991 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266790 got 227633266789 for osd.2 2026-03-31T21:09:16.009 INFO:teuthology.orchestra.run.vm00.stdout:163208757359 2026-03-31T21:09:16.018 INFO:teuthology.orchestra.run.vm00.stdout:206158430312 2026-03-31T21:09:16.020 INFO:teuthology.orchestra.run.vm00.stdout:90194313366 2026-03-31T21:09:16.027 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757360 got 163208757359 for osd.0 2026-03-31T21:09:16.033 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313366 got 90194313366 for osd.1 2026-03-31T21:09:16.033 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:16.035 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430312 got 206158430312 for osd.3 2026-03-31T21:09:16.035 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:16.992 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:17.027 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:17.183 INFO:teuthology.orchestra.run.vm00.stdout:227633266790 2026-03-31T21:09:17.194 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266790 got 227633266790 for osd.2 2026-03-31T21:09:17.194 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:17.223 INFO:teuthology.orchestra.run.vm00.stdout:163208757361 2026-03-31T21:09:17.234 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757360 got 163208757361 for osd.0 2026-03-31T21:09:17.234 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:17.234 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:17.455 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:17.455 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:17.465 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2026-03-31T21:09:15.853990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10575,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16281,"num_bytes_recovered":133647536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":315632,"kb_used_data":120120,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282799888,"statfs":{"total":289910292480,"available":289587085312,"internally_reserved":0,"allocated":123002880,"data_stored":122129026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-148,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":46124,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:09:14.454935+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:09:14.454935+0000","last_peered":"2026-03-31T21:09:14.454935+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:09:14.454935+0000","last_undegraded":"2026-03-31T21:09:14.454935+0000","last_fullsized":"2026-03-31T21:09:14.454935+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10575,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16279,"num_bytes_recovered":133319344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10575,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16279,"num_bytes_recovered":133319344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":175685632,"data_stored":175685632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430312,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266790,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757361,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41464,"kb_used_data":11944,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94330376,"statfs":{"total":96636764160,"available":96594305024,"internally_reserved":0,"allocated":12230656,"data_stored":11936982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11440128,"data_stored":11440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:17.465 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 46124, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:09:14.454935+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:09:14.454935+0000', 'last_peered': '2026-03-31T21:09:14.454935+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:09:14.454935+0000', 'last_undegraded': '2026-03-31T21:09:14.454935+0000', 'last_fullsized': '2026-03-31T21:09:14.454935+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10575, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16279, 'num_bytes_recovered': 133319344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:17.465 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:09:22.466 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:09:22.466 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:09:22.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:09:22.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:09:22.565 INFO:teuthology.orchestra.run.vm00.stdout:90194313368 2026-03-31T21:09:22.566 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:22.570 INFO:teuthology.orchestra.run.vm00.stdout:227633266793 2026-03-31T21:09:22.570 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:22.571 INFO:teuthology.orchestra.run.vm00.stdout:163208757363 2026-03-31T21:09:22.571 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:22.579 INFO:teuthology.orchestra.run.vm00.stdout:206158430314 2026-03-31T21:09:22.579 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:22.776 INFO:teuthology.orchestra.run.vm00.stdout:163208757362 2026-03-31T21:09:22.776 INFO:teuthology.orchestra.run.vm00.stdout:90194313367 2026-03-31T21:09:22.789 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313368 got 90194313367 for osd.1 2026-03-31T21:09:22.792 INFO:teuthology.orchestra.run.vm00.stdout:227633266792 2026-03-31T21:09:22.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757363 got 163208757362 for osd.0 2026-03-31T21:09:22.805 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266793 got 227633266792 for osd.2 2026-03-31T21:09:22.835 INFO:teuthology.orchestra.run.vm00.stdout:206158430313 2026-03-31T21:09:22.845 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430314 got 206158430313 for osd.3 2026-03-31T21:09:23.790 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:23.794 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:23.806 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:23.846 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:24.003 INFO:teuthology.orchestra.run.vm00.stdout:90194313367 2026-03-31T21:09:24.019 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313368 got 90194313367 for osd.1 2026-03-31T21:09:24.041 INFO:teuthology.orchestra.run.vm00.stdout:163208757363 2026-03-31T21:09:24.043 INFO:teuthology.orchestra.run.vm00.stdout:227633266793 2026-03-31T21:09:24.054 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757363 got 163208757363 for osd.0 2026-03-31T21:09:24.055 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:24.057 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266793 got 227633266793 for osd.2 2026-03-31T21:09:24.057 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:24.100 INFO:teuthology.orchestra.run.vm00.stdout:206158430314 2026-03-31T21:09:24.111 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430314 got 206158430314 for osd.3 2026-03-31T21:09:24.111 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:25.019 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:25.208 INFO:teuthology.orchestra.run.vm00.stdout:90194313369 2026-03-31T21:09:25.218 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313368 got 90194313369 for osd.1 2026-03-31T21:09:25.218 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:25.218 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:25.400 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:25.400 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:25.410 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2026-03-31T21:09:23.854879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10496,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16360,"num_bytes_recovered":134294704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":315944,"kb_used_data":120432,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282799576,"statfs":{"total":289910292480,"available":289586765824,"internally_reserved":0,"allocated":123322368,"data_stored":122448514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-129,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":1056768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001379"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":46282,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:09:22.482565+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:09:22.482565+0000","last_peered":"2026-03-31T21:09:22.482565+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:09:22.482565+0000","last_undegraded":"2026-03-31T21:09:22.482565+0000","last_fullsized":"2026-03-31T21:09:22.482565+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10496,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16358,"num_bytes_recovered":133966512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10496,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16358,"num_bytes_recovered":133966512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176009216,"data_stored":176009216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430314,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266793,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313369,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41776,"kb_used_data":12256,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94330064,"statfs":{"total":96636764160,"available":96593985536,"internally_reserved":0,"allocated":12550144,"data_stored":12256470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":11763712,"data_stored":11763712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:25.410 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 46282, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:09:22.482565+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:09:22.482565+0000', 'last_peered': '2026-03-31T21:09:22.482565+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:09:22.482565+0000', 'last_undegraded': '2026-03-31T21:09:22.482565+0000', 'last_fullsized': '2026-03-31T21:09:22.482565+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10496, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16358, 'num_bytes_recovered': 133966512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:25.410 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:09:30.411 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:09:30.412 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:09:30.412 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:09:30.412 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:09:30.519 INFO:teuthology.orchestra.run.vm00.stdout:227633266795 2026-03-31T21:09:30.519 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:30.520 INFO:teuthology.orchestra.run.vm00.stdout:90194313371 2026-03-31T21:09:30.520 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:30.521 INFO:teuthology.orchestra.run.vm00.stdout:163208757366 2026-03-31T21:09:30.521 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:30.529 INFO:teuthology.orchestra.run.vm00.stdout:206158430317 2026-03-31T21:09:30.529 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:30.728 INFO:teuthology.orchestra.run.vm00.stdout:163208757365 2026-03-31T21:09:30.730 INFO:teuthology.orchestra.run.vm00.stdout:90194313370 2026-03-31T21:09:30.731 INFO:teuthology.orchestra.run.vm00.stdout:227633266794 2026-03-31T21:09:30.740 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757366 got 163208757365 for osd.0 2026-03-31T21:09:30.742 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266795 got 227633266794 for osd.2 2026-03-31T21:09:30.749 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313371 got 90194313370 for osd.1 2026-03-31T21:09:30.777 INFO:teuthology.orchestra.run.vm00.stdout:206158430316 2026-03-31T21:09:30.787 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430317 got 206158430316 for osd.3 2026-03-31T21:09:31.741 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:31.742 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:31.750 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:31.788 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:31.939 INFO:teuthology.orchestra.run.vm00.stdout:163208757365 2026-03-31T21:09:31.956 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757366 got 163208757365 for osd.0 2026-03-31T21:09:31.960 INFO:teuthology.orchestra.run.vm00.stdout:227633266794 2026-03-31T21:09:31.971 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266795 got 227633266794 for osd.2 2026-03-31T21:09:32.003 INFO:teuthology.orchestra.run.vm00.stdout:90194313370 2026-03-31T21:09:32.016 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313371 got 90194313370 for osd.1 2026-03-31T21:09:32.041 INFO:teuthology.orchestra.run.vm00.stdout:206158430316 2026-03-31T21:09:32.051 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430317 got 206158430316 for osd.3 2026-03-31T21:09:32.956 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:32.972 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:33.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:33.052 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:33.152 INFO:teuthology.orchestra.run.vm00.stdout:163208757366 2026-03-31T21:09:33.165 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757366 got 163208757366 for osd.0 2026-03-31T21:09:33.165 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:33.201 INFO:teuthology.orchestra.run.vm00.stdout:227633266796 2026-03-31T21:09:33.215 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266795 got 227633266796 for osd.2 2026-03-31T21:09:33.215 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:33.279 INFO:teuthology.orchestra.run.vm00.stdout:90194313371 2026-03-31T21:09:33.293 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313371 got 90194313371 for osd.1 2026-03-31T21:09:33.293 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:33.293 INFO:teuthology.orchestra.run.vm00.stdout:206158430317 2026-03-31T21:09:33.304 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430317 got 206158430317 for osd.3 2026-03-31T21:09:33.304 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:33.304 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:33.532 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:33.532 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:33.542 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2026-03-31T21:09:31.855736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10419,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16437,"num_bytes_recovered":134925488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":316248,"kb_used_data":120736,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282799272,"statfs":{"total":289910292480,"available":289586454528,"internally_reserved":0,"allocated":123633664,"data_stored":122759810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001290"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":46436,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:09:30.510528+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:09:30.510528+0000","last_peered":"2026-03-31T21:09:30.510528+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:09:30.510528+0000","last_undegraded":"2026-03-31T21:09:30.510528+0000","last_fullsized":"2026-03-31T21:09:30.510528+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10419,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16435,"num_bytes_recovered":134597296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":51,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:05:36.796195+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:05:36.796195+0000","last_peered":"2026-03-31T21:05:36.796195+0000","last_clean":"2026-03-31T21:05:36.796195+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:05:36.796195+0000","last_undegraded":"2026-03-31T21:05:36.796195+0000","last_fullsized":"2026-03-31T21:05:36.796195+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10419,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16435,"num_bytes_recovered":134597296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176320512,"data_stored":176320512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430317,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266796,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757366,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42080,"kb_used_data":12560,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94329760,"statfs":{"total":96636764160,"available":96593674240,"internally_reserved":0,"allocated":12861440,"data_stored":12567766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12075008,"data_stored":12075008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:33.543 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 46436, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:09:30.510528+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:09:30.510528+0000', 'last_peered': '2026-03-31T21:09:30.510528+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:09:30.510528+0000', 'last_undegraded': '2026-03-31T21:09:30.510528+0000', 'last_fullsized': '2026-03-31T21:09:30.510528+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10419, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16435, 'num_bytes_recovered': 134597296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:33.543 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:09:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:09:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:09:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:09:38.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:09:38.647 INFO:teuthology.orchestra.run.vm00.stdout:227633266798 2026-03-31T21:09:38.648 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:38.656 INFO:teuthology.orchestra.run.vm00.stdout:90194313374 2026-03-31T21:09:38.656 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:38.664 INFO:teuthology.orchestra.run.vm00.stdout:163208757369 2026-03-31T21:09:38.664 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:38.666 INFO:teuthology.orchestra.run.vm00.stdout:206158430319 2026-03-31T21:09:38.666 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:38.869 INFO:teuthology.orchestra.run.vm00.stdout:227633266797 2026-03-31T21:09:38.872 INFO:teuthology.orchestra.run.vm00.stdout:163208757368 2026-03-31T21:09:38.876 INFO:teuthology.orchestra.run.vm00.stdout:90194313373 2026-03-31T21:09:38.881 INFO:teuthology.orchestra.run.vm00.stdout:206158430318 2026-03-31T21:09:38.883 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266798 got 227633266797 for osd.2 2026-03-31T21:09:38.889 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757369 got 163208757368 for osd.0 2026-03-31T21:09:38.890 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313374 got 90194313373 for osd.1 2026-03-31T21:09:38.898 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430319 got 206158430318 for osd.3 2026-03-31T21:09:39.884 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:39.890 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:39.891 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:39.898 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:40.096 INFO:teuthology.orchestra.run.vm00.stdout:227633266797 2026-03-31T21:09:40.114 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266798 got 227633266797 for osd.2 2026-03-31T21:09:40.134 INFO:teuthology.orchestra.run.vm00.stdout:206158430320 2026-03-31T21:09:40.138 INFO:teuthology.orchestra.run.vm00.stdout:163208757370 2026-03-31T21:09:40.142 INFO:teuthology.orchestra.run.vm00.stdout:90194313374 2026-03-31T21:09:40.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430319 got 206158430320 for osd.3 2026-03-31T21:09:40.146 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:40.152 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757369 got 163208757370 for osd.0 2026-03-31T21:09:40.152 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:40.153 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313374 got 90194313374 for osd.1 2026-03-31T21:09:40.153 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:41.115 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:41.302 INFO:teuthology.orchestra.run.vm00.stdout:227633266798 2026-03-31T21:09:41.312 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266798 got 227633266798 for osd.2 2026-03-31T21:09:41.312 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:41.312 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:41.583 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:41.583 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:41.594 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2026-03-31T21:09:39.856712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10332,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16524,"num_bytes_recovered":135638192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":316608,"kb_used_data":121096,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282798912,"statfs":{"total":289910292480,"available":289586085888,"internally_reserved":0,"allocated":124002304,"data_stored":123128450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001398"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":46610,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:09:39.348447+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:09:39.348447+0000","last_peered":"2026-03-31T21:09:39.348447+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:09:39.348447+0000","last_undegraded":"2026-03-31T21:09:39.348447+0000","last_fullsized":"2026-03-31T21:09:39.348447+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10332,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":135310000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10332,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16522,"num_bytes_recovered":135310000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176697344,"data_stored":176697344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430320,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757370,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42440,"kb_used_data":12920,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94329400,"statfs":{"total":96636764160,"available":96593305600,"internally_reserved":0,"allocated":13230080,"data_stored":12936406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12451840,"data_stored":12451840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:41.594 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 46610, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:09:39.348447+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:09:39.348447+0000', 'last_peered': '2026-03-31T21:09:39.348447+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:09:39.348447+0000', 'last_undegraded': '2026-03-31T21:09:39.348447+0000', 'last_fullsized': '2026-03-31T21:09:39.348447+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10332, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16522, 'num_bytes_recovered': 135310000, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:41.594 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:09:46.595 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:09:46.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:09:46.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:09:46.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:09:46.703 INFO:teuthology.orchestra.run.vm00.stdout:90194313376 2026-03-31T21:09:46.703 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:46.704 INFO:teuthology.orchestra.run.vm00.stdout:227633266800 2026-03-31T21:09:46.704 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:46.709 INFO:teuthology.orchestra.run.vm00.stdout:206158430322 2026-03-31T21:09:46.710 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:46.712 INFO:teuthology.orchestra.run.vm00.stdout:163208757372 2026-03-31T21:09:46.712 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:46.913 INFO:teuthology.orchestra.run.vm00.stdout:90194313375 2026-03-31T21:09:46.914 INFO:teuthology.orchestra.run.vm00.stdout:227633266799 2026-03-31T21:09:46.915 INFO:teuthology.orchestra.run.vm00.stdout:206158430321 2026-03-31T21:09:46.927 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313376 got 90194313375 for osd.1 2026-03-31T21:09:46.928 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266800 got 227633266799 for osd.2 2026-03-31T21:09:46.929 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430322 got 206158430321 for osd.3 2026-03-31T21:09:46.981 INFO:teuthology.orchestra.run.vm00.stdout:163208757371 2026-03-31T21:09:46.992 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757372 got 163208757371 for osd.0 2026-03-31T21:09:47.927 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:47.929 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:47.929 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:47.993 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:48.130 INFO:teuthology.orchestra.run.vm00.stdout:90194313375 2026-03-31T21:09:48.133 INFO:teuthology.orchestra.run.vm00.stdout:227633266799 2026-03-31T21:09:48.136 INFO:teuthology.orchestra.run.vm00.stdout:206158430321 2026-03-31T21:09:48.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313376 got 90194313375 for osd.1 2026-03-31T21:09:48.147 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266800 got 227633266799 for osd.2 2026-03-31T21:09:48.152 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430322 got 206158430321 for osd.3 2026-03-31T21:09:48.204 INFO:teuthology.orchestra.run.vm00.stdout:163208757372 2026-03-31T21:09:48.215 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757372 got 163208757372 for osd.0 2026-03-31T21:09:48.215 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:49.148 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:49.148 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:49.153 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:49.344 INFO:teuthology.orchestra.run.vm00.stdout:90194313377 2026-03-31T21:09:49.349 INFO:teuthology.orchestra.run.vm00.stdout:227633266801 2026-03-31T21:09:49.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313376 got 90194313377 for osd.1 2026-03-31T21:09:49.355 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:49.366 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266800 got 227633266801 for osd.2 2026-03-31T21:09:49.366 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:49.390 INFO:teuthology.orchestra.run.vm00.stdout:206158430322 2026-03-31T21:09:49.401 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430322 got 206158430322 for osd.3 2026-03-31T21:09:49.401 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:49.401 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:49.624 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:49.624 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:49.633 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2026-03-31T21:09:47.857733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10262,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16594,"num_bytes_recovered":136211632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":316876,"kb_used_data":121364,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282798644,"statfs":{"total":289910292480,"available":289585811456,"internally_reserved":0,"allocated":124276736,"data_stored":123402882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-119,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001483"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":46750,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:09:46.667981+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:09:46.667981+0000","last_peered":"2026-03-31T21:09:46.667981+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:09:46.667981+0000","last_undegraded":"2026-03-31T21:09:46.667981+0000","last_fullsized":"2026-03-31T21:09:46.667981+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10262,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16592,"num_bytes_recovered":135883440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10262,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16592,"num_bytes_recovered":135883440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":176967680,"data_stored":176967680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430322,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42708,"kb_used_data":13188,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94329132,"statfs":{"total":96636764160,"available":96593031168,"internally_reserved":0,"allocated":13504512,"data_stored":13210838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12722176,"data_stored":12722176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:49.633 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 46750, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:09:46.667981+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:09:46.667981+0000', 'last_peered': '2026-03-31T21:09:46.667981+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:09:46.667981+0000', 'last_undegraded': '2026-03-31T21:09:46.667981+0000', 'last_fullsized': '2026-03-31T21:09:46.667981+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10262, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16592, 'num_bytes_recovered': 135883440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:49.634 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:09:54.635 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:09:54.635 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:09:54.635 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:09:54.635 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:09:54.737 INFO:teuthology.orchestra.run.vm00.stdout:227633266803 2026-03-31T21:09:54.737 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:54.738 INFO:teuthology.orchestra.run.vm00.stdout:90194313379 2026-03-31T21:09:54.738 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:54.743 INFO:teuthology.orchestra.run.vm00.stdout:163208757374 2026-03-31T21:09:54.743 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:54.746 INFO:teuthology.orchestra.run.vm00.stdout:206158430325 2026-03-31T21:09:54.746 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:54.946 INFO:teuthology.orchestra.run.vm00.stdout:227633266802 2026-03-31T21:09:54.951 INFO:teuthology.orchestra.run.vm00.stdout:90194313378 2026-03-31T21:09:54.957 INFO:teuthology.orchestra.run.vm00.stdout:163208757373 2026-03-31T21:09:54.960 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266803 got 227633266802 for osd.2 2026-03-31T21:09:54.969 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313379 got 90194313378 for osd.1 2026-03-31T21:09:54.975 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757374 got 163208757373 for osd.0 2026-03-31T21:09:54.998 INFO:teuthology.orchestra.run.vm00.stdout:206158430323 2026-03-31T21:09:55.010 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430325 got 206158430323 for osd.3 2026-03-31T21:09:55.961 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:09:55.970 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:09:55.976 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:09:56.011 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:09:56.200 INFO:teuthology.orchestra.run.vm00.stdout:163208757375 2026-03-31T21:09:56.210 INFO:teuthology.orchestra.run.vm00.stdout:90194313379 2026-03-31T21:09:56.211 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757374 got 163208757375 for osd.0 2026-03-31T21:09:56.211 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:56.221 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313379 got 90194313379 for osd.1 2026-03-31T21:09:56.221 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:56.231 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:09:56.229+0000 7fd96f7fe640 0 --2- 192.168.123.100:0/4093364163 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7fd97007f480 0x7fd970171fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:09:56.246 INFO:teuthology.orchestra.run.vm00.stdout:206158430325 2026-03-31T21:09:56.257 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430325 got 206158430325 for osd.3 2026-03-31T21:09:56.257 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:56.355 INFO:teuthology.orchestra.run.vm00.stdout:227633266803 2026-03-31T21:09:56.364 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266803 got 227633266803 for osd.2 2026-03-31T21:09:56.364 DEBUG:teuthology.parallel:result is None 2026-03-31T21:09:56.364 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:09:56.548 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:09:56.548 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:09:56.558 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2026-03-31T21:09:55.858707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10183,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16673,"num_bytes_recovered":136858800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":317192,"kb_used_data":121680,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282798328,"statfs":{"total":289910292480,"available":289585487872,"internally_reserved":0,"allocated":124600320,"data_stored":123726466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-149,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001469"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":46908,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:09:54.704156+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:09:54.704156+0000","last_peered":"2026-03-31T21:09:54.704156+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:09:54.704156+0000","last_undegraded":"2026-03-31T21:09:54.704156+0000","last_fullsized":"2026-03-31T21:09:54.704156+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10183,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16671,"num_bytes_recovered":136530608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10183,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16671,"num_bytes_recovered":136530608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":177291264,"data_stored":177291264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430325,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266803,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313379,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43024,"kb_used_data":13504,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94328816,"statfs":{"total":96636764160,"available":96592707584,"internally_reserved":0,"allocated":13828096,"data_stored":13534422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13045760,"data_stored":13045760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:09:56.558 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 46908, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:09:54.704156+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:09:54.704156+0000', 'last_peered': '2026-03-31T21:09:54.704156+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:09:54.704156+0000', 'last_undegraded': '2026-03-31T21:09:54.704156+0000', 'last_fullsized': '2026-03-31T21:09:54.704156+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10183, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16671, 'num_bytes_recovered': 136530608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:09:56.558 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:01.558 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:01.559 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:01.559 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:01.559 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:01.657 INFO:teuthology.orchestra.run.vm00.stdout:90194313381 2026-03-31T21:10:01.657 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:01.661 INFO:teuthology.orchestra.run.vm00.stdout:163208757377 2026-03-31T21:10:01.661 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:01.670 INFO:teuthology.orchestra.run.vm00.stdout:227633266805 2026-03-31T21:10:01.670 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:01.674 INFO:teuthology.orchestra.run.vm00.stdout:206158430327 2026-03-31T21:10:01.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:01.874 INFO:teuthology.orchestra.run.vm00.stdout:90194313380 2026-03-31T21:10:01.881 INFO:teuthology.orchestra.run.vm00.stdout:227633266804 2026-03-31T21:10:01.881 INFO:teuthology.orchestra.run.vm00.stdout:163208757376 2026-03-31T21:10:01.884 INFO:teuthology.orchestra.run.vm00.stdout:206158430326 2026-03-31T21:10:01.887 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313381 got 90194313380 for osd.1 2026-03-31T21:10:01.895 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266805 got 227633266804 for osd.2 2026-03-31T21:10:01.895 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757377 got 163208757376 for osd.0 2026-03-31T21:10:01.899 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430327 got 206158430326 for osd.3 2026-03-31T21:10:02.888 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:02.896 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:02.896 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:02.899 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:03.107 INFO:teuthology.orchestra.run.vm00.stdout:90194313381 2026-03-31T21:10:03.126 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313381 got 90194313381 for osd.1 2026-03-31T21:10:03.126 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:03.141 INFO:teuthology.orchestra.run.vm00.stdout:227633266806 2026-03-31T21:10:03.142 INFO:teuthology.orchestra.run.vm00.stdout:163208757377 2026-03-31T21:10:03.149 INFO:teuthology.orchestra.run.vm00.stdout:206158430327 2026-03-31T21:10:03.154 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266805 got 227633266806 for osd.2 2026-03-31T21:10:03.154 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:03.157 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757377 got 163208757377 for osd.0 2026-03-31T21:10:03.157 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:03.163 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430327 got 206158430327 for osd.3 2026-03-31T21:10:03.163 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:03.163 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:03.394 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:03.395 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:03.404 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2026-03-31T21:10:01.859383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10115,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16741,"num_bytes_recovered":137415856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":317456,"kb_used_data":121944,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282798064,"statfs":{"total":289910292480,"available":289585217536,"internally_reserved":0,"allocated":124870656,"data_stored":123996802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-120,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001388"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":47045,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:01.635443+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:01.635443+0000","last_peered":"2026-03-31T21:10:01.635443+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:01.635443+0000","last_undegraded":"2026-03-31T21:10:01.635443+0000","last_fullsized":"2026-03-31T21:10:01.635443+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10115,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16739,"num_bytes_recovered":137087664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10115,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16739,"num_bytes_recovered":137087664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":177569792,"data_stored":177569792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430327,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313381,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757377,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43288,"kb_used_data":13768,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94328552,"statfs":{"total":96636764160,"available":96592437248,"internally_reserved":0,"allocated":14098432,"data_stored":13804758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13324288,"data_stored":13324288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:03.405 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 47045, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:01.635443+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:01.635443+0000', 'last_peered': '2026-03-31T21:10:01.635443+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:01.635443+0000', 'last_undegraded': '2026-03-31T21:10:01.635443+0000', 'last_fullsized': '2026-03-31T21:10:01.635443+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10115, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16739, 'num_bytes_recovered': 137087664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:03.405 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:08.405 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:08.406 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:08.406 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:08.406 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:08.510 INFO:teuthology.orchestra.run.vm00.stdout:90194313384 2026-03-31T21:10:08.510 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:08.510 INFO:teuthology.orchestra.run.vm00.stdout:206158430329 2026-03-31T21:10:08.510 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:08.515 INFO:teuthology.orchestra.run.vm00.stdout:163208757379 2026-03-31T21:10:08.515 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:08.517 INFO:teuthology.orchestra.run.vm00.stdout:227633266808 2026-03-31T21:10:08.517 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:08.741 INFO:teuthology.orchestra.run.vm00.stdout:206158430328 2026-03-31T21:10:08.743 INFO:teuthology.orchestra.run.vm00.stdout:90194313383 2026-03-31T21:10:08.747 INFO:teuthology.orchestra.run.vm00.stdout:227633266807 2026-03-31T21:10:08.758 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430329 got 206158430328 for osd.3 2026-03-31T21:10:08.758 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313384 got 90194313383 for osd.1 2026-03-31T21:10:08.764 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266808 got 227633266807 for osd.2 2026-03-31T21:10:08.766 INFO:teuthology.orchestra.run.vm00.stdout:163208757378 2026-03-31T21:10:08.776 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757379 got 163208757378 for osd.0 2026-03-31T21:10:09.759 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:09.759 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:09.764 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:09.776 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:09.957 INFO:teuthology.orchestra.run.vm00.stdout:90194313383 2026-03-31T21:10:09.975 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313384 got 90194313383 for osd.1 2026-03-31T21:10:09.977 INFO:teuthology.orchestra.run.vm00.stdout:206158430328 2026-03-31T21:10:09.989 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430329 got 206158430328 for osd.3 2026-03-31T21:10:10.011 INFO:teuthology.orchestra.run.vm00.stdout:163208757378 2026-03-31T21:10:10.011 INFO:teuthology.orchestra.run.vm00.stdout:227633266807 2026-03-31T21:10:10.023 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266808 got 227633266807 for osd.2 2026-03-31T21:10:10.023 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757379 got 163208757378 for osd.0 2026-03-31T21:10:10.976 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:10.990 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:11.024 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:11.024 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:11.190 INFO:teuthology.orchestra.run.vm00.stdout:90194313384 2026-03-31T21:10:11.207 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313384 got 90194313384 for osd.1 2026-03-31T21:10:11.207 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:11.234 INFO:teuthology.orchestra.run.vm00.stdout:206158430330 2026-03-31T21:10:11.253 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430329 got 206158430330 for osd.3 2026-03-31T21:10:11.253 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:11.272 INFO:teuthology.orchestra.run.vm00.stdout:227633266808 2026-03-31T21:10:11.275 INFO:teuthology.orchestra.run.vm00.stdout:163208757380 2026-03-31T21:10:11.285 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266808 got 227633266808 for osd.2 2026-03-31T21:10:11.286 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:11.293 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757379 got 163208757380 for osd.0 2026-03-31T21:10:11.293 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:11.293 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:11.523 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:11.523 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:11.534 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2026-03-31T21:10:09.860445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10039,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16817,"num_bytes_recovered":138038448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":317772,"kb_used_data":122260,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282797748,"statfs":{"total":289910292480,"available":289584893952,"internally_reserved":0,"allocated":125194240,"data_stored":124320386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":47197,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:09.374941+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:09.374941+0000","last_peered":"2026-03-31T21:10:09.374941+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:09.374941+0000","last_undegraded":"2026-03-31T21:10:09.374941+0000","last_fullsized":"2026-03-31T21:10:09.374941+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10039,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16815,"num_bytes_recovered":137710256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10039,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16815,"num_bytes_recovered":137710256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":177893376,"data_stored":177893376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430330,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757380,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43604,"kb_used_data":14084,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94328236,"statfs":{"total":96636764160,"available":96592113664,"internally_reserved":0,"allocated":14422016,"data_stored":14128342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13647872,"data_stored":13647872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:11.535 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 47197, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:09.374941+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:09.374941+0000', 'last_peered': '2026-03-31T21:10:09.374941+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:09.374941+0000', 'last_undegraded': '2026-03-31T21:10:09.374941+0000', 'last_fullsized': '2026-03-31T21:10:09.374941+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10039, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16815, 'num_bytes_recovered': 137710256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:11.535 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:16.536 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:16.536 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:16.536 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:16.536 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:16.638 INFO:teuthology.orchestra.run.vm00.stdout:227633266810 2026-03-31T21:10:16.638 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:16.639 INFO:teuthology.orchestra.run.vm00.stdout:206158430332 2026-03-31T21:10:16.639 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:16.639 INFO:teuthology.orchestra.run.vm00.stdout:163208757382 2026-03-31T21:10:16.639 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:16.647 INFO:teuthology.orchestra.run.vm00.stdout:90194313386 2026-03-31T21:10:16.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:16.844 INFO:teuthology.orchestra.run.vm00.stdout:163208757381 2026-03-31T21:10:16.845 INFO:teuthology.orchestra.run.vm00.stdout:206158430331 2026-03-31T21:10:16.846 INFO:teuthology.orchestra.run.vm00.stdout:227633266809 2026-03-31T21:10:16.858 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266810 got 227633266809 for osd.2 2026-03-31T21:10:16.860 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430332 got 206158430331 for osd.3 2026-03-31T21:10:16.862 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757382 got 163208757381 for osd.0 2026-03-31T21:10:16.919 INFO:teuthology.orchestra.run.vm00.stdout:90194313385 2026-03-31T21:10:16.933 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313386 got 90194313385 for osd.1 2026-03-31T21:10:17.859 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:17.860 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:17.863 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:17.935 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:18.082 INFO:teuthology.orchestra.run.vm00.stdout:227633266811 2026-03-31T21:10:18.083 INFO:teuthology.orchestra.run.vm00.stdout:206158430332 2026-03-31T21:10:18.094 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430332 got 206158430332 for osd.3 2026-03-31T21:10:18.094 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:18.094 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266810 got 227633266811 for osd.2 2026-03-31T21:10:18.094 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:18.115 INFO:teuthology.orchestra.run.vm00.stdout:163208757382 2026-03-31T21:10:18.126 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757382 got 163208757382 for osd.0 2026-03-31T21:10:18.126 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:18.155 INFO:teuthology.orchestra.run.vm00.stdout:90194313387 2026-03-31T21:10:18.166 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313386 got 90194313387 for osd.1 2026-03-31T21:10:18.166 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:18.166 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:18.389 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:18.389 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:18.398 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2026-03-31T21:10:17.861555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9968,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16888,"num_bytes_recovered":138620080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":318052,"kb_used_data":122540,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282797468,"statfs":{"total":289910292480,"available":289584607232,"internally_reserved":0,"allocated":125480960,"data_stored":124607106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-120,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001614"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":47339,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:16.607978+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:16.607978+0000","last_peered":"2026-03-31T21:10:16.607978+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:16.607978+0000","last_undegraded":"2026-03-31T21:10:16.607978+0000","last_fullsized":"2026-03-31T21:10:16.607978+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9968,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16886,"num_bytes_recovered":138291888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9968,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16886,"num_bytes_recovered":138291888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178171904,"data_stored":178171904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430332,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757382,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43884,"kb_used_data":14364,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94327956,"statfs":{"total":96636764160,"available":96591826944,"internally_reserved":0,"allocated":14708736,"data_stored":14415062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13926400,"data_stored":13926400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:18.399 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 47339, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:16.607978+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:16.607978+0000', 'last_peered': '2026-03-31T21:10:16.607978+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:16.607978+0000', 'last_undegraded': '2026-03-31T21:10:16.607978+0000', 'last_fullsized': '2026-03-31T21:10:16.607978+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9968, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16886, 'num_bytes_recovered': 138291888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:18.399 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:23.400 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:23.400 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:23.400 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:23.400 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:23.509 INFO:teuthology.orchestra.run.vm00.stdout:227633266813 2026-03-31T21:10:23.509 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:23.509 INFO:teuthology.orchestra.run.vm00.stdout:90194313389 2026-03-31T21:10:23.509 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:23.517 INFO:teuthology.orchestra.run.vm00.stdout:206158430334 2026-03-31T21:10:23.517 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:23.517 INFO:teuthology.orchestra.run.vm00.stdout:163208757384 2026-03-31T21:10:23.517 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:23.715 INFO:teuthology.orchestra.run.vm00.stdout:163208757383 2026-03-31T21:10:23.718 INFO:teuthology.orchestra.run.vm00.stdout:227633266812 2026-03-31T21:10:23.727 INFO:teuthology.orchestra.run.vm00.stdout:206158430333 2026-03-31T21:10:23.727 INFO:teuthology.orchestra.run.vm00.stdout:90194313387 2026-03-31T21:10:23.730 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757384 got 163208757383 for osd.0 2026-03-31T21:10:23.736 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266813 got 227633266812 for osd.2 2026-03-31T21:10:23.739 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313389 got 90194313387 for osd.1 2026-03-31T21:10:23.745 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430334 got 206158430333 for osd.3 2026-03-31T21:10:24.731 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:24.737 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:24.740 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:24.746 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:24.950 INFO:teuthology.orchestra.run.vm00.stdout:163208757384 2026-03-31T21:10:24.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757384 got 163208757384 for osd.0 2026-03-31T21:10:24.973 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:24.983 INFO:teuthology.orchestra.run.vm00.stdout:206158430334 2026-03-31T21:10:24.984 INFO:teuthology.orchestra.run.vm00.stdout:90194313389 2026-03-31T21:10:24.985 INFO:teuthology.orchestra.run.vm00.stdout:227633266813 2026-03-31T21:10:24.997 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430334 got 206158430334 for osd.3 2026-03-31T21:10:24.997 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:25.000 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313389 got 90194313389 for osd.1 2026-03-31T21:10:25.000 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:25.002 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266813 got 227633266813 for osd.2 2026-03-31T21:10:25.002 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:25.002 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:25.224 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:25.224 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:25.234 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2026-03-31T21:10:23.862309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9902,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16954,"num_bytes_recovered":139160752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":318320,"kb_used_data":122808,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282797200,"statfs":{"total":289910292480,"available":289584332800,"internally_reserved":0,"allocated":125755392,"data_stored":124881538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-137,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001624"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":47471,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:23.434191+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:23.434191+0000","last_peered":"2026-03-31T21:10:23.434191+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:23.434191+0000","last_undegraded":"2026-03-31T21:10:23.434191+0000","last_fullsized":"2026-03-31T21:10:23.434191+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9902,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16952,"num_bytes_recovered":138832560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9902,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16952,"num_bytes_recovered":138832560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178442240,"data_stored":178442240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313389,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757384,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44152,"kb_used_data":14632,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94327688,"statfs":{"total":96636764160,"available":96591552512,"internally_reserved":0,"allocated":14983168,"data_stored":14689494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14196736,"data_stored":14196736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:25.234 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 47471, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:23.434191+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:23.434191+0000', 'last_peered': '2026-03-31T21:10:23.434191+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:23.434191+0000', 'last_undegraded': '2026-03-31T21:10:23.434191+0000', 'last_fullsized': '2026-03-31T21:10:23.434191+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9902, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 16952, 'num_bytes_recovered': 138832560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:25.234 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:30.236 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:30.236 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:30.236 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:30.236 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:30.342 INFO:teuthology.orchestra.run.vm00.stdout:90194313391 2026-03-31T21:10:30.343 INFO:teuthology.orchestra.run.vm00.stdout:227633266815 2026-03-31T21:10:30.343 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:30.343 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:30.344 INFO:teuthology.orchestra.run.vm00.stdout:206158430337 2026-03-31T21:10:30.344 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:30.346 INFO:teuthology.orchestra.run.vm00.stdout:163208757387 2026-03-31T21:10:30.346 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:30.544 INFO:teuthology.orchestra.run.vm00.stdout:206158430336 2026-03-31T21:10:30.554 INFO:teuthology.orchestra.run.vm00.stdout:90194313390 2026-03-31T21:10:30.557 INFO:teuthology.orchestra.run.vm00.stdout:227633266814 2026-03-31T21:10:30.568 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430337 got 206158430336 for osd.3 2026-03-31T21:10:30.570 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313391 got 90194313390 for osd.1 2026-03-31T21:10:30.570 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266815 got 227633266814 for osd.2 2026-03-31T21:10:30.612 INFO:teuthology.orchestra.run.vm00.stdout:163208757386 2026-03-31T21:10:30.622 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757387 got 163208757386 for osd.0 2026-03-31T21:10:31.569 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:31.571 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:31.571 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:31.623 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:31.787 INFO:teuthology.orchestra.run.vm00.stdout:90194313390 2026-03-31T21:10:31.788 INFO:teuthology.orchestra.run.vm00.stdout:227633266814 2026-03-31T21:10:31.791 INFO:teuthology.orchestra.run.vm00.stdout:206158430336 2026-03-31T21:10:31.802 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313391 got 90194313390 for osd.1 2026-03-31T21:10:31.806 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266815 got 227633266814 for osd.2 2026-03-31T21:10:31.808 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430337 got 206158430336 for osd.3 2026-03-31T21:10:31.856 INFO:teuthology.orchestra.run.vm00.stdout:163208757386 2026-03-31T21:10:31.867 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757387 got 163208757386 for osd.0 2026-03-31T21:10:32.802 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:32.806 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:32.809 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:32.867 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:33.012 INFO:teuthology.orchestra.run.vm00.stdout:90194313391 2026-03-31T21:10:33.032 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313391 got 90194313391 for osd.1 2026-03-31T21:10:33.032 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:33.059 INFO:teuthology.orchestra.run.vm00.stdout:206158430337 2026-03-31T21:10:33.070 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430337 got 206158430337 for osd.3 2026-03-31T21:10:33.070 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:33.091 INFO:teuthology.orchestra.run.vm00.stdout:163208757387 2026-03-31T21:10:33.103 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757387 got 163208757387 for osd.0 2026-03-31T21:10:33.103 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:33.122 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:10:33.119+0000 7ff2727fc640 0 --2- 192.168.123.100:0/2646037494 >> v2:192.168.123.100:3300/0 conn(0x7ff274154ce0 0x7ff2741750c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:10:33.243 INFO:teuthology.orchestra.run.vm00.stdout:227633266816 2026-03-31T21:10:33.255 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266815 got 227633266816 for osd.2 2026-03-31T21:10:33.255 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:33.255 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:33.486 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:33.486 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:33.496 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2026-03-31T21:10:31.863292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9835,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17021,"num_bytes_recovered":139709616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":318592,"kb_used_data":123080,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282796928,"statfs":{"total":289910292480,"available":289584054272,"internally_reserved":0,"allocated":126033920,"data_stored":125160066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-106,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001489"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":47605,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:30.256418+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:30.256418+0000","last_peered":"2026-03-31T21:10:30.256418+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:30.256418+0000","last_undegraded":"2026-03-31T21:10:30.256418+0000","last_fullsized":"2026-03-31T21:10:30.256418+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9835,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17019,"num_bytes_recovered":139381424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9835,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17019,"num_bytes_recovered":139381424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":178716672,"data_stored":178716672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430337,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313391,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757387,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44424,"kb_used_data":14904,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94327416,"statfs":{"total":96636764160,"available":96591273984,"internally_reserved":0,"allocated":15261696,"data_stored":14968022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14471168,"data_stored":14471168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:33.496 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 47605, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:30.256418+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:30.256418+0000', 'last_peered': '2026-03-31T21:10:30.256418+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:30.256418+0000', 'last_undegraded': '2026-03-31T21:10:30.256418+0000', 'last_fullsized': '2026-03-31T21:10:30.256418+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9835, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17019, 'num_bytes_recovered': 139381424, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:33.496 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:38.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:38.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:38.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:38.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:38.609 INFO:teuthology.orchestra.run.vm00.stdout:206158430339 2026-03-31T21:10:38.609 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:38.610 INFO:teuthology.orchestra.run.vm00.stdout:163208757389 2026-03-31T21:10:38.610 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:38.616 INFO:teuthology.orchestra.run.vm00.stdout:90194313394 2026-03-31T21:10:38.616 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:38.619 INFO:teuthology.orchestra.run.vm00.stdout:227633266818 2026-03-31T21:10:38.619 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:38.818 INFO:teuthology.orchestra.run.vm00.stdout:90194313393 2026-03-31T21:10:38.825 INFO:teuthology.orchestra.run.vm00.stdout:163208757388 2026-03-31T21:10:38.834 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313394 got 90194313393 for osd.1 2026-03-31T21:10:38.842 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757389 got 163208757388 for osd.0 2026-03-31T21:10:38.844 INFO:teuthology.orchestra.run.vm00.stdout:206158430338 2026-03-31T21:10:38.857 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430339 got 206158430338 for osd.3 2026-03-31T21:10:38.880 INFO:teuthology.orchestra.run.vm00.stdout:227633266817 2026-03-31T21:10:38.891 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266818 got 227633266817 for osd.2 2026-03-31T21:10:39.834 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:39.843 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:39.857 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:39.892 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:40.060 INFO:teuthology.orchestra.run.vm00.stdout:90194313394 2026-03-31T21:10:40.076 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313394 got 90194313394 for osd.1 2026-03-31T21:10:40.076 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:40.105 INFO:teuthology.orchestra.run.vm00.stdout:206158430340 2026-03-31T21:10:40.112 INFO:teuthology.orchestra.run.vm00.stdout:163208757390 2026-03-31T21:10:40.118 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430339 got 206158430340 for osd.3 2026-03-31T21:10:40.118 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:40.125 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757389 got 163208757390 for osd.0 2026-03-31T21:10:40.125 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:40.156 INFO:teuthology.orchestra.run.vm00.stdout:227633266818 2026-03-31T21:10:40.168 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266818 got 227633266818 for osd.2 2026-03-31T21:10:40.168 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:40.168 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:40.402 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:40.402 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:40.413 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2026-03-31T21:10:39.864297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9745,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17111,"num_bytes_recovered":140446896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":318948,"kb_used_data":123436,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282796572,"statfs":{"total":289910292480,"available":289583689728,"internally_reserved":0,"allocated":126398464,"data_stored":125524610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-148,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":47785,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:39.430325+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:39.430325+0000","last_peered":"2026-03-31T21:10:39.430325+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:39.430325+0000","last_undegraded":"2026-03-31T21:10:39.430325+0000","last_fullsized":"2026-03-31T21:10:39.430325+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9745,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17109,"num_bytes_recovered":140118704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9745,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17109,"num_bytes_recovered":140118704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179097600,"data_stored":179097600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430340,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757390,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44780,"kb_used_data":15260,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94327060,"statfs":{"total":96636764160,"available":96590909440,"internally_reserved":0,"allocated":15626240,"data_stored":15332566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14852096,"data_stored":14852096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:40.413 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 47785, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:39.430325+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:39.430325+0000', 'last_peered': '2026-03-31T21:10:39.430325+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:39.430325+0000', 'last_undegraded': '2026-03-31T21:10:39.430325+0000', 'last_fullsized': '2026-03-31T21:10:39.430325+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9745, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17109, 'num_bytes_recovered': 140118704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:40.413 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:45.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:45.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:45.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:45.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:45.516 INFO:teuthology.orchestra.run.vm00.stdout:90194313396 2026-03-31T21:10:45.516 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:45.524 INFO:teuthology.orchestra.run.vm00.stdout:227633266820 2026-03-31T21:10:45.524 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:45.525 INFO:teuthology.orchestra.run.vm00.stdout:163208757392 2026-03-31T21:10:45.525 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:45.531 INFO:teuthology.orchestra.run.vm00.stdout:206158430342 2026-03-31T21:10:45.531 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:45.725 INFO:teuthology.orchestra.run.vm00.stdout:90194313395 2026-03-31T21:10:45.731 INFO:teuthology.orchestra.run.vm00.stdout:163208757390 2026-03-31T21:10:45.737 INFO:teuthology.orchestra.run.vm00.stdout:227633266819 2026-03-31T21:10:45.740 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313396 got 90194313395 for osd.1 2026-03-31T21:10:45.747 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757392 got 163208757390 for osd.0 2026-03-31T21:10:45.752 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266820 got 227633266819 for osd.2 2026-03-31T21:10:45.793 INFO:teuthology.orchestra.run.vm00.stdout:206158430340 2026-03-31T21:10:45.804 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430342 got 206158430340 for osd.3 2026-03-31T21:10:46.741 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:46.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:46.753 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:46.804 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:46.940 INFO:teuthology.orchestra.run.vm00.stdout:90194313396 2026-03-31T21:10:46.955 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313396 got 90194313396 for osd.1 2026-03-31T21:10:46.955 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:46.994 INFO:teuthology.orchestra.run.vm00.stdout:163208757392 2026-03-31T21:10:46.998 INFO:teuthology.orchestra.run.vm00.stdout:227633266820 2026-03-31T21:10:47.007 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757392 got 163208757392 for osd.0 2026-03-31T21:10:47.007 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:47.015 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266820 got 227633266820 for osd.2 2026-03-31T21:10:47.015 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:47.051 INFO:teuthology.orchestra.run.vm00.stdout:206158430342 2026-03-31T21:10:47.063 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430342 got 206158430342 for osd.3 2026-03-31T21:10:47.063 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:47.063 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:47.287 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:47.287 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:47.299 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2026-03-31T21:10:45.865085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9686,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17170,"num_bytes_recovered":140930224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":319184,"kb_used_data":123672,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282796336,"statfs":{"total":289910292480,"available":289583448064,"internally_reserved":0,"allocated":126640128,"data_stored":125766274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-149,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1220608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001532"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":47903,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:45.451751+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:45.451751+0000","last_peered":"2026-03-31T21:10:45.451751+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:45.451751+0000","last_undegraded":"2026-03-31T21:10:45.451751+0000","last_fullsized":"2026-03-31T21:10:45.451751+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9686,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17168,"num_bytes_recovered":140602032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9686,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17168,"num_bytes_recovered":140602032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179326976,"data_stored":179326976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430342,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757392,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45016,"kb_used_data":15496,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94326824,"statfs":{"total":96636764160,"available":96590667776,"internally_reserved":0,"allocated":15867904,"data_stored":15574230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15081472,"data_stored":15081472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:47.299 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 47903, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:45.451751+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:45.451751+0000', 'last_peered': '2026-03-31T21:10:45.451751+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:45.451751+0000', 'last_undegraded': '2026-03-31T21:10:45.451751+0000', 'last_fullsized': '2026-03-31T21:10:45.451751+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9686, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17168, 'num_bytes_recovered': 140602032, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:47.299 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:10:52.300 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:10:52.305 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:10:52.305 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:10:52.305 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:10:52.418 INFO:teuthology.orchestra.run.vm00.stdout:90194313398 2026-03-31T21:10:52.418 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:52.423 INFO:teuthology.orchestra.run.vm00.stdout:227633266823 2026-03-31T21:10:52.423 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:52.426 INFO:teuthology.orchestra.run.vm00.stdout:206158430344 2026-03-31T21:10:52.427 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:52.620 INFO:teuthology.orchestra.run.vm00.stdout:163208757394 2026-03-31T21:10:52.620 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:52.638 INFO:teuthology.orchestra.run.vm00.stdout:90194313397 2026-03-31T21:10:52.650 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313398 got 90194313397 for osd.1 2026-03-31T21:10:52.669 INFO:teuthology.orchestra.run.vm00.stdout:206158430343 2026-03-31T21:10:52.676 INFO:teuthology.orchestra.run.vm00.stdout:227633266822 2026-03-31T21:10:52.683 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430344 got 206158430343 for osd.3 2026-03-31T21:10:52.688 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266823 got 227633266822 for osd.2 2026-03-31T21:10:52.830 INFO:teuthology.orchestra.run.vm00.stdout:163208757393 2026-03-31T21:10:52.841 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757394 got 163208757393 for osd.0 2026-03-31T21:10:53.651 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:53.684 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:53.688 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:53.841 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:53.851 INFO:teuthology.orchestra.run.vm00.stdout:90194313397 2026-03-31T21:10:53.869 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313398 got 90194313397 for osd.1 2026-03-31T21:10:53.891 INFO:teuthology.orchestra.run.vm00.stdout:227633266822 2026-03-31T21:10:53.892 INFO:teuthology.orchestra.run.vm00.stdout:206158430343 2026-03-31T21:10:53.904 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266823 got 227633266822 for osd.2 2026-03-31T21:10:53.913 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430344 got 206158430343 for osd.3 2026-03-31T21:10:54.034 INFO:teuthology.orchestra.run.vm00.stdout:163208757393 2026-03-31T21:10:54.044 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757394 got 163208757393 for osd.0 2026-03-31T21:10:54.870 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:10:54.905 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:10:54.913 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:10:55.044 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:10:55.065 INFO:teuthology.orchestra.run.vm00.stdout:90194313399 2026-03-31T21:10:55.083 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313398 got 90194313399 for osd.1 2026-03-31T21:10:55.083 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:55.113 INFO:teuthology.orchestra.run.vm00.stdout:206158430344 2026-03-31T21:10:55.115 INFO:teuthology.orchestra.run.vm00.stdout:227633266823 2026-03-31T21:10:55.127 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430344 got 206158430344 for osd.3 2026-03-31T21:10:55.127 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:55.128 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266823 got 227633266823 for osd.2 2026-03-31T21:10:55.128 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:55.252 INFO:teuthology.orchestra.run.vm00.stdout:163208757394 2026-03-31T21:10:55.263 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757394 got 163208757394 for osd.0 2026-03-31T21:10:55.263 DEBUG:teuthology.parallel:result is None 2026-03-31T21:10:55.263 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:10:55.451 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:10:55.451 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:10:55.462 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2026-03-31T21:10:53.866126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9619,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17237,"num_bytes_recovered":141479088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":319456,"kb_used_data":123944,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282796064,"statfs":{"total":289910292480,"available":289583169536,"internally_reserved":0,"allocated":126918656,"data_stored":126044802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001546"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":48037,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:10:52.392960+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:10:52.392960+0000","last_peered":"2026-03-31T21:10:52.392960+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:10:52.392960+0000","last_undegraded":"2026-03-31T21:10:52.392960+0000","last_fullsized":"2026-03-31T21:10:52.392960+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9619,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17235,"num_bytes_recovered":141150896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9619,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17235,"num_bytes_recovered":141150896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179609600,"data_stored":179609600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430344,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757394,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45288,"kb_used_data":15768,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94326552,"statfs":{"total":96636764160,"available":96590389248,"internally_reserved":0,"allocated":16146432,"data_stored":15852758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15364096,"data_stored":15364096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:10:55.462 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 48037, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:10:52.392960+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:10:52.392960+0000', 'last_peered': '2026-03-31T21:10:52.392960+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:10:52.392960+0000', 'last_undegraded': '2026-03-31T21:10:52.392960+0000', 'last_fullsized': '2026-03-31T21:10:52.392960+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9619, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17235, 'num_bytes_recovered': 141150896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:10:55.463 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:00.463 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:00.464 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:00.464 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:00.464 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:00.564 INFO:teuthology.orchestra.run.vm00.stdout:163208757397 2026-03-31T21:11:00.565 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:00.570 INFO:teuthology.orchestra.run.vm00.stdout:206158430347 2026-03-31T21:11:00.570 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:00.571 INFO:teuthology.orchestra.run.vm00.stdout:227633266825 2026-03-31T21:11:00.571 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:00.580 INFO:teuthology.orchestra.run.vm00.stdout:90194313401 2026-03-31T21:11:00.580 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:00.771 INFO:teuthology.orchestra.run.vm00.stdout:163208757396 2026-03-31T21:11:00.774 INFO:teuthology.orchestra.run.vm00.stdout:227633266824 2026-03-31T21:11:00.788 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266825 got 227633266824 for osd.2 2026-03-31T21:11:00.789 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757397 got 163208757396 for osd.0 2026-03-31T21:11:00.795 INFO:teuthology.orchestra.run.vm00.stdout:206158430346 2026-03-31T21:11:00.808 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430347 got 206158430346 for osd.3 2026-03-31T21:11:00.843 INFO:teuthology.orchestra.run.vm00.stdout:90194313400 2026-03-31T21:11:00.853 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313401 got 90194313400 for osd.1 2026-03-31T21:11:01.789 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:01.790 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:01.809 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:01.853 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:01.996 INFO:teuthology.orchestra.run.vm00.stdout:227633266826 2026-03-31T21:11:02.008 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266825 got 227633266826 for osd.2 2026-03-31T21:11:02.008 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:02.045 INFO:teuthology.orchestra.run.vm00.stdout:163208757397 2026-03-31T21:11:02.058 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757397 got 163208757397 for osd.0 2026-03-31T21:11:02.058 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:02.090 INFO:teuthology.orchestra.run.vm00.stdout:90194313401 2026-03-31T21:11:02.090 INFO:teuthology.orchestra.run.vm00.stdout:206158430347 2026-03-31T21:11:02.101 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430347 got 206158430347 for osd.3 2026-03-31T21:11:02.102 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:02.102 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313401 got 90194313401 for osd.1 2026-03-31T21:11:02.102 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:02.102 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:02.323 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:02.323 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:02.332 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2026-03-31T21:11:01.866974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9540,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17316,"num_bytes_recovered":142126256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":319768,"kb_used_data":124256,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282795752,"statfs":{"total":289910292480,"available":289582850048,"internally_reserved":0,"allocated":127238144,"data_stored":126364290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001294"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":48195,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:00.486286+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:00.486286+0000","last_peered":"2026-03-31T21:11:00.486286+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:00.486286+0000","last_undegraded":"2026-03-31T21:11:00.486286+0000","last_fullsized":"2026-03-31T21:11:00.486286+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9540,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17314,"num_bytes_recovered":141798064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9540,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17314,"num_bytes_recovered":141798064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":179924992,"data_stored":179924992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430347,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313401,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757397,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45600,"kb_used_data":16080,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94326240,"statfs":{"total":96636764160,"available":96590069760,"internally_reserved":0,"allocated":16465920,"data_stored":16172246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15679488,"data_stored":15679488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:02.333 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 48195, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:00.486286+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:00.486286+0000', 'last_peered': '2026-03-31T21:11:00.486286+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:00.486286+0000', 'last_undegraded': '2026-03-31T21:11:00.486286+0000', 'last_fullsized': '2026-03-31T21:11:00.486286+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9540, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17314, 'num_bytes_recovered': 141798064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:02.333 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:07.334 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:07.334 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:07.334 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:07.334 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:07.439 INFO:teuthology.orchestra.run.vm00.stdout:206158430349 2026-03-31T21:11:07.439 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:07.439 INFO:teuthology.orchestra.run.vm00.stdout:163208757399 2026-03-31T21:11:07.439 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:07.440 INFO:teuthology.orchestra.run.vm00.stdout:227633266828 2026-03-31T21:11:07.440 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:07.446 INFO:teuthology.orchestra.run.vm00.stdout:90194313403 2026-03-31T21:11:07.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:07.651 INFO:teuthology.orchestra.run.vm00.stdout:206158430348 2026-03-31T21:11:07.655 INFO:teuthology.orchestra.run.vm00.stdout:227633266826 2026-03-31T21:11:07.659 INFO:teuthology.orchestra.run.vm00.stdout:163208757398 2026-03-31T21:11:07.672 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430349 got 206158430348 for osd.3 2026-03-31T21:11:07.676 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757399 got 163208757398 for osd.0 2026-03-31T21:11:07.679 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266828 got 227633266826 for osd.2 2026-03-31T21:11:07.709 INFO:teuthology.orchestra.run.vm00.stdout:90194313402 2026-03-31T21:11:07.720 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313403 got 90194313402 for osd.1 2026-03-31T21:11:08.672 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:08.677 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:08.679 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:08.720 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:08.879 INFO:teuthology.orchestra.run.vm00.stdout:206158430349 2026-03-31T21:11:08.895 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430349 got 206158430349 for osd.3 2026-03-31T21:11:08.895 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:08.918 INFO:teuthology.orchestra.run.vm00.stdout:163208757399 2026-03-31T21:11:08.927 INFO:teuthology.orchestra.run.vm00.stdout:227633266828 2026-03-31T21:11:08.933 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757399 got 163208757399 for osd.0 2026-03-31T21:11:08.933 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:08.941 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266828 got 227633266828 for osd.2 2026-03-31T21:11:08.941 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:08.977 INFO:teuthology.orchestra.run.vm00.stdout:90194313404 2026-03-31T21:11:08.989 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313403 got 90194313404 for osd.1 2026-03-31T21:11:08.989 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:08.989 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:09.214 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:09.214 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:09.223 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2026-03-31T21:11:07.867747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9472,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17384,"num_bytes_recovered":142683312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":320044,"kb_used_data":124532,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282795476,"statfs":{"total":289910292480,"available":289582567424,"internally_reserved":0,"allocated":127520768,"data_stored":126646914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001391"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":48331,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:07.412583+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:07.412583+0000","last_peered":"2026-03-31T21:11:07.412583+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:07.412583+0000","last_undegraded":"2026-03-31T21:11:07.412583+0000","last_fullsized":"2026-03-31T21:11:07.412583+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9472,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17382,"num_bytes_recovered":142355120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9472,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17382,"num_bytes_recovered":142355120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180203520,"data_stored":180203520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430349,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757399,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45876,"kb_used_data":16356,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94325964,"statfs":{"total":96636764160,"available":96589787136,"internally_reserved":0,"allocated":16748544,"data_stored":16454870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":15958016,"data_stored":15958016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:09.224 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 48331, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:07.412583+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:07.412583+0000', 'last_peered': '2026-03-31T21:11:07.412583+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:07.412583+0000', 'last_undegraded': '2026-03-31T21:11:07.412583+0000', 'last_fullsized': '2026-03-31T21:11:07.412583+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9472, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17382, 'num_bytes_recovered': 142355120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:09.224 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:14.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:14.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:14.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:14.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:14.336 INFO:teuthology.orchestra.run.vm00.stdout:90194313406 2026-03-31T21:11:14.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:14.336 INFO:teuthology.orchestra.run.vm00.stdout:206158430351 2026-03-31T21:11:14.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:14.336 INFO:teuthology.orchestra.run.vm00.stdout:163208757401 2026-03-31T21:11:14.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:14.531 INFO:teuthology.orchestra.run.vm00.stdout:227633266830 2026-03-31T21:11:14.532 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:14.583 INFO:teuthology.orchestra.run.vm00.stdout:90194313405 2026-03-31T21:11:14.583 INFO:teuthology.orchestra.run.vm00.stdout:206158430350 2026-03-31T21:11:14.583 INFO:teuthology.orchestra.run.vm00.stdout:163208757400 2026-03-31T21:11:14.599 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430351 got 206158430350 for osd.3 2026-03-31T21:11:14.600 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313406 got 90194313405 for osd.1 2026-03-31T21:11:14.600 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757401 got 163208757400 for osd.0 2026-03-31T21:11:14.740 INFO:teuthology.orchestra.run.vm00.stdout:227633266829 2026-03-31T21:11:14.751 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266830 got 227633266829 for osd.2 2026-03-31T21:11:15.600 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:15.600 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:15.601 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:15.752 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:15.820 INFO:teuthology.orchestra.run.vm00.stdout:163208757400 2026-03-31T21:11:15.821 INFO:teuthology.orchestra.run.vm00.stdout:206158430350 2026-03-31T21:11:15.833 INFO:teuthology.orchestra.run.vm00.stdout:90194313405 2026-03-31T21:11:15.835 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430351 got 206158430350 for osd.3 2026-03-31T21:11:15.843 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757401 got 163208757400 for osd.0 2026-03-31T21:11:15.854 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313406 got 90194313405 for osd.1 2026-03-31T21:11:15.973 INFO:teuthology.orchestra.run.vm00.stdout:227633266829 2026-03-31T21:11:15.985 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266830 got 227633266829 for osd.2 2026-03-31T21:11:16.836 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:16.844 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:16.855 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:16.986 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:17.047 INFO:teuthology.orchestra.run.vm00.stdout:206158430352 2026-03-31T21:11:17.065 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430351 got 206158430352 for osd.3 2026-03-31T21:11:17.065 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:17.093 INFO:teuthology.orchestra.run.vm00.stdout:90194313406 2026-03-31T21:11:17.105 INFO:teuthology.orchestra.run.vm00.stdout:163208757402 2026-03-31T21:11:17.110 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313406 got 90194313406 for osd.1 2026-03-31T21:11:17.110 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:17.121 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757401 got 163208757402 for osd.0 2026-03-31T21:11:17.121 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:17.198 INFO:teuthology.orchestra.run.vm00.stdout:227633266830 2026-03-31T21:11:17.209 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266830 got 227633266830 for osd.2 2026-03-31T21:11:17.209 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:17.209 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:17.408 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:17.409 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:17.422 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2026-03-31T21:11:15.868805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9404,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17452,"num_bytes_recovered":143240368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":320316,"kb_used_data":124804,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282795204,"statfs":{"total":289910292480,"available":289582288896,"internally_reserved":0,"allocated":127799296,"data_stored":126925442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-136,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001592"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":48467,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:14.429918+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:14.429918+0000","last_peered":"2026-03-31T21:11:14.429918+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:14.429918+0000","last_undegraded":"2026-03-31T21:11:14.429918+0000","last_fullsized":"2026-03-31T21:11:14.429918+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9404,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17450,"num_bytes_recovered":142912176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9404,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17450,"num_bytes_recovered":142912176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180494336,"data_stored":180494336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46148,"kb_used_data":16628,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94325692,"statfs":{"total":96636764160,"available":96589508608,"internally_reserved":0,"allocated":17027072,"data_stored":16733398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16248832,"data_stored":16248832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:17.422 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 48467, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:14.429918+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:14.429918+0000', 'last_peered': '2026-03-31T21:11:14.429918+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:14.429918+0000', 'last_undegraded': '2026-03-31T21:11:14.429918+0000', 'last_fullsized': '2026-03-31T21:11:14.429918+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9404, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17450, 'num_bytes_recovered': 142912176, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:17.422 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:22.423 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:22.423 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:22.424 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:22.424 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:22.535 INFO:teuthology.orchestra.run.vm00.stdout:90194313408 2026-03-31T21:11:22.535 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:22.535 INFO:teuthology.orchestra.run.vm00.stdout:206158430354 2026-03-31T21:11:22.535 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:22.539 INFO:teuthology.orchestra.run.vm00.stdout:227633266833 2026-03-31T21:11:22.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:22.540 INFO:teuthology.orchestra.run.vm00.stdout:163208757404 2026-03-31T21:11:22.540 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:22.741 INFO:teuthology.orchestra.run.vm00.stdout:206158430353 2026-03-31T21:11:22.749 INFO:teuthology.orchestra.run.vm00.stdout:90194313407 2026-03-31T21:11:22.758 INFO:teuthology.orchestra.run.vm00.stdout:227633266832 2026-03-31T21:11:22.760 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430354 got 206158430353 for osd.3 2026-03-31T21:11:22.764 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313408 got 90194313407 for osd.1 2026-03-31T21:11:22.770 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266833 got 227633266832 for osd.2 2026-03-31T21:11:22.801 INFO:teuthology.orchestra.run.vm00.stdout:163208757403 2026-03-31T21:11:22.813 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757404 got 163208757403 for osd.0 2026-03-31T21:11:23.761 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:23.765 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:23.771 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:23.814 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:23.978 INFO:teuthology.orchestra.run.vm00.stdout:206158430353 2026-03-31T21:11:23.996 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430354 got 206158430353 for osd.3 2026-03-31T21:11:24.020 INFO:teuthology.orchestra.run.vm00.stdout:227633266832 2026-03-31T21:11:24.034 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266833 got 227633266832 for osd.2 2026-03-31T21:11:24.041 INFO:teuthology.orchestra.run.vm00.stdout:90194313409 2026-03-31T21:11:24.057 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313408 got 90194313409 for osd.1 2026-03-31T21:11:24.057 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:24.092 INFO:teuthology.orchestra.run.vm00.stdout:163208757404 2026-03-31T21:11:24.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757404 got 163208757404 for osd.0 2026-03-31T21:11:24.104 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:24.997 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:25.035 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:25.225 INFO:teuthology.orchestra.run.vm00.stdout:206158430354 2026-03-31T21:11:25.237 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430354 got 206158430354 for osd.3 2026-03-31T21:11:25.237 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:25.253 INFO:teuthology.orchestra.run.vm00.stdout:227633266833 2026-03-31T21:11:25.264 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266833 got 227633266833 for osd.2 2026-03-31T21:11:25.264 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:25.264 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:25.498 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:25.498 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:25.510 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2026-03-31T21:11:23.869844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9325,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17531,"num_bytes_recovered":143887536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":320624,"kb_used_data":125112,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282794896,"statfs":{"total":289910292480,"available":289581973504,"internally_reserved":0,"allocated":128114688,"data_stored":127240834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-128,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":48625,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:22.515861+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:22.515861+0000","last_peered":"2026-03-31T21:11:22.515861+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:22.515861+0000","last_undegraded":"2026-03-31T21:11:22.515861+0000","last_fullsized":"2026-03-31T21:11:22.515861+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9325,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17529,"num_bytes_recovered":143559344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9325,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17529,"num_bytes_recovered":143559344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180805632,"data_stored":180805632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430354,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757404,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46456,"kb_used_data":16936,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94325384,"statfs":{"total":96636764160,"available":96589193216,"internally_reserved":0,"allocated":17342464,"data_stored":17048790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16560128,"data_stored":16560128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:25.510 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 48625, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:22.515861+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:22.515861+0000', 'last_peered': '2026-03-31T21:11:22.515861+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:22.515861+0000', 'last_undegraded': '2026-03-31T21:11:22.515861+0000', 'last_fullsized': '2026-03-31T21:11:22.515861+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9325, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17529, 'num_bytes_recovered': 143559344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:25.510 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:30.511 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:30.511 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:30.511 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:30.512 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:30.624 INFO:teuthology.orchestra.run.vm00.stdout:227633266835 2026-03-31T21:11:30.625 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:30.630 INFO:teuthology.orchestra.run.vm00.stdout:90194313411 2026-03-31T21:11:30.630 INFO:teuthology.orchestra.run.vm00.stdout:206158430357 2026-03-31T21:11:30.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:30.631 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:30.632 INFO:teuthology.orchestra.run.vm00.stdout:163208757407 2026-03-31T21:11:30.632 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:30.844 INFO:teuthology.orchestra.run.vm00.stdout:227633266834 2026-03-31T21:11:30.859 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266835 got 227633266834 for osd.2 2026-03-31T21:11:30.889 INFO:teuthology.orchestra.run.vm00.stdout:206158430356 2026-03-31T21:11:30.904 INFO:teuthology.orchestra.run.vm00.stdout:90194313410 2026-03-31T21:11:30.908 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430357 got 206158430356 for osd.3 2026-03-31T21:11:30.917 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313411 got 90194313410 for osd.1 2026-03-31T21:11:30.957 INFO:teuthology.orchestra.run.vm00.stdout:163208757406 2026-03-31T21:11:30.969 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757407 got 163208757406 for osd.0 2026-03-31T21:11:31.861 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:31.909 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:31.918 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:31.969 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:32.080 INFO:teuthology.orchestra.run.vm00.stdout:227633266836 2026-03-31T21:11:32.096 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266835 got 227633266836 for osd.2 2026-03-31T21:11:32.096 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:32.131 INFO:teuthology.orchestra.run.vm00.stdout:206158430357 2026-03-31T21:11:32.145 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430357 got 206158430357 for osd.3 2026-03-31T21:11:32.145 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:32.216 INFO:teuthology.orchestra.run.vm00.stdout:163208757407 2026-03-31T21:11:32.228 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757407 got 163208757407 for osd.0 2026-03-31T21:11:32.228 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:32.247 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:11:32.244+0000 7fad17fff640 0 --2- 192.168.123.100:0/1014803487 >> [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] conn(0x7fad1807ed30 0x7fad1818f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:11:32.377 INFO:teuthology.orchestra.run.vm00.stdout:90194313411 2026-03-31T21:11:32.389 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313411 got 90194313411 for osd.1 2026-03-31T21:11:32.389 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:32.389 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:32.633 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:32.633 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:32.646 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2026-03-31T21:11:31.871007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9249,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17607,"num_bytes_recovered":144510128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":320932,"kb_used_data":125420,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282794588,"statfs":{"total":289910292480,"available":289581658112,"internally_reserved":0,"allocated":128430080,"data_stored":127556226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-107,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":876544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001688"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":48777,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:30.555052+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:30.555052+0000","last_peered":"2026-03-31T21:11:30.555052+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:30.555052+0000","last_undegraded":"2026-03-31T21:11:30.555052+0000","last_fullsized":"2026-03-31T21:11:30.555052+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9249,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17605,"num_bytes_recovered":144181936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":52,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:07:37.551996+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:07:37.551996+0000","last_peered":"2026-03-31T21:07:37.551996+0000","last_clean":"2026-03-31T21:07:37.551996+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:07:37.551996+0000","last_undegraded":"2026-03-31T21:07:37.551996+0000","last_fullsized":"2026-03-31T21:07:37.551996+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9249,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17605,"num_bytes_recovered":144181936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181116928,"data_stored":181116928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313411,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757407,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46764,"kb_used_data":17244,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94325076,"statfs":{"total":96636764160,"available":96588877824,"internally_reserved":0,"allocated":17657856,"data_stored":17364182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16871424,"data_stored":16871424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:32.647 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 48777, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:30.555052+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:30.555052+0000', 'last_peered': '2026-03-31T21:11:30.555052+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:30.555052+0000', 'last_undegraded': '2026-03-31T21:11:30.555052+0000', 'last_fullsized': '2026-03-31T21:11:30.555052+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9249, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17605, 'num_bytes_recovered': 144181936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:32.647 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:37.648 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:37.648 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:37.648 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:37.648 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:37.780 INFO:teuthology.orchestra.run.vm00.stdout:227633266838 2026-03-31T21:11:37.780 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:37.781 INFO:teuthology.orchestra.run.vm00.stdout:90194313414 2026-03-31T21:11:37.781 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:37.782 INFO:teuthology.orchestra.run.vm00.stdout:163208757409 2026-03-31T21:11:37.783 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:37.788 INFO:teuthology.orchestra.run.vm00.stdout:206158430359 2026-03-31T21:11:37.788 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:38.001 INFO:teuthology.orchestra.run.vm00.stdout:90194313412 2026-03-31T21:11:38.001 INFO:teuthology.orchestra.run.vm00.stdout:163208757408 2026-03-31T21:11:38.014 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313414 got 90194313412 for osd.1 2026-03-31T21:11:38.015 INFO:teuthology.orchestra.run.vm00.stdout:227633266836 2026-03-31T21:11:38.020 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757409 got 163208757408 for osd.0 2026-03-31T21:11:38.028 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266838 got 227633266836 for osd.2 2026-03-31T21:11:38.057 INFO:teuthology.orchestra.run.vm00.stdout:206158430358 2026-03-31T21:11:38.067 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430359 got 206158430358 for osd.3 2026-03-31T21:11:39.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:39.021 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:39.030 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:39.068 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:39.243 INFO:teuthology.orchestra.run.vm00.stdout:90194313414 2026-03-31T21:11:39.260 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313414 got 90194313414 for osd.1 2026-03-31T21:11:39.260 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:39.273 INFO:teuthology.orchestra.run.vm00.stdout:227633266838 2026-03-31T21:11:39.291 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266838 got 227633266838 for osd.2 2026-03-31T21:11:39.291 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:39.298 INFO:teuthology.orchestra.run.vm00.stdout:163208757409 2026-03-31T21:11:39.310 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757409 got 163208757409 for osd.0 2026-03-31T21:11:39.310 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:39.357 INFO:teuthology.orchestra.run.vm00.stdout:206158430359 2026-03-31T21:11:39.369 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430359 got 206158430359 for osd.3 2026-03-31T21:11:39.369 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:39.369 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:39.564 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:39.564 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:39.576 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":372,"stamp":"2026-03-31T21:11:37.871857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9179,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17677,"num_bytes_recovered":145083568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":321200,"kb_used_data":125688,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282794320,"statfs":{"total":289910292480,"available":289581383680,"internally_reserved":0,"allocated":128704512,"data_stored":127830658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-128,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001738"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":48917,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:37.770160+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:37.770160+0000","last_peered":"2026-03-31T21:11:37.770160+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:37.770160+0000","last_undegraded":"2026-03-31T21:11:37.770160+0000","last_fullsized":"2026-03-31T21:11:37.770160+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9179,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17675,"num_bytes_recovered":144755376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9179,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17675,"num_bytes_recovered":144755376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181399552,"data_stored":181399552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430359,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757409,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47032,"kb_used_data":17512,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94324808,"statfs":{"total":96636764160,"available":96588603392,"internally_reserved":0,"allocated":17932288,"data_stored":17638614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17154048,"data_stored":17154048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:39.576 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 48917, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:37.770160+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:37.770160+0000', 'last_peered': '2026-03-31T21:11:37.770160+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:37.770160+0000', 'last_undegraded': '2026-03-31T21:11:37.770160+0000', 'last_fullsized': '2026-03-31T21:11:37.770160+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9179, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17675, 'num_bytes_recovered': 144755376, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:39.576 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:44.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:44.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:44.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:44.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:44.690 INFO:teuthology.orchestra.run.vm00.stdout:163208757411 2026-03-31T21:11:44.690 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:44.698 INFO:teuthology.orchestra.run.vm00.stdout:206158430362 2026-03-31T21:11:44.698 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:44.708 INFO:teuthology.orchestra.run.vm00.stdout:90194313416 2026-03-31T21:11:44.708 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:44.711 INFO:teuthology.orchestra.run.vm00.stdout:227633266840 2026-03-31T21:11:44.712 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:44.927 INFO:teuthology.orchestra.run.vm00.stdout:163208757410 2026-03-31T21:11:44.940 INFO:teuthology.orchestra.run.vm00.stdout:90194313415 2026-03-31T21:11:44.949 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757411 got 163208757410 for osd.0 2026-03-31T21:11:44.953 INFO:teuthology.orchestra.run.vm00.stdout:206158430360 2026-03-31T21:11:44.953 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313416 got 90194313415 for osd.1 2026-03-31T21:11:44.966 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430362 got 206158430360 for osd.3 2026-03-31T21:11:44.976 INFO:teuthology.orchestra.run.vm00.stdout:227633266839 2026-03-31T21:11:44.988 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266840 got 227633266839 for osd.2 2026-03-31T21:11:45.950 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:45.954 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:45.967 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:45.989 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:46.183 INFO:teuthology.orchestra.run.vm00.stdout:163208757410 2026-03-31T21:11:46.197 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757411 got 163208757410 for osd.0 2026-03-31T21:11:46.223 INFO:teuthology.orchestra.run.vm00.stdout:90194313416 2026-03-31T21:11:46.226 INFO:teuthology.orchestra.run.vm00.stdout:206158430362 2026-03-31T21:11:46.229 INFO:teuthology.orchestra.run.vm00.stdout:227633266840 2026-03-31T21:11:46.238 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313416 got 90194313416 for osd.1 2026-03-31T21:11:46.238 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:46.241 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430362 got 206158430362 for osd.3 2026-03-31T21:11:46.241 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:46.248 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266840 got 227633266840 for osd.2 2026-03-31T21:11:46.248 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:47.198 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:47.398 INFO:teuthology.orchestra.run.vm00.stdout:163208757412 2026-03-31T21:11:47.410 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757411 got 163208757412 for osd.0 2026-03-31T21:11:47.410 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:47.410 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:47.605 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:47.605 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:47.617 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2026-03-31T21:11:45.872750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9113,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17743,"num_bytes_recovered":145624240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":321468,"kb_used_data":125956,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282794052,"statfs":{"total":289910292480,"available":289581109248,"internally_reserved":0,"allocated":128978944,"data_stored":128105090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-136,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001458"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":49049,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:44.649403+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:44.649403+0000","last_peered":"2026-03-31T21:11:44.649403+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:44.649403+0000","last_undegraded":"2026-03-31T21:11:44.649403+0000","last_fullsized":"2026-03-31T21:11:44.649403+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9113,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17741,"num_bytes_recovered":145296048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9113,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17741,"num_bytes_recovered":145296048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181678080,"data_stored":181678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430362,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313416,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757412,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47300,"kb_used_data":17780,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94324540,"statfs":{"total":96636764160,"available":96588328960,"internally_reserved":0,"allocated":18206720,"data_stored":17913046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17432576,"data_stored":17432576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:47.617 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 49049, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:44.649403+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:44.649403+0000', 'last_peered': '2026-03-31T21:11:44.649403+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:44.649403+0000', 'last_undegraded': '2026-03-31T21:11:44.649403+0000', 'last_fullsized': '2026-03-31T21:11:44.649403+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9113, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17741, 'num_bytes_recovered': 145296048, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:47.617 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:52.618 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:52.619 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:52.619 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:52.619 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:52.730 INFO:teuthology.orchestra.run.vm00.stdout:206158430364 2026-03-31T21:11:52.730 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:52.730 INFO:teuthology.orchestra.run.vm00.stdout:227633266843 2026-03-31T21:11:52.730 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:52.730 INFO:teuthology.orchestra.run.vm00.stdout:163208757414 2026-03-31T21:11:52.730 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:52.734 INFO:teuthology.orchestra.run.vm00.stdout:90194313419 2026-03-31T21:11:52.735 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:52.950 INFO:teuthology.orchestra.run.vm00.stdout:206158430363 2026-03-31T21:11:52.950 INFO:teuthology.orchestra.run.vm00.stdout:227633266842 2026-03-31T21:11:52.957 INFO:teuthology.orchestra.run.vm00.stdout:90194313417 2026-03-31T21:11:52.959 INFO:teuthology.orchestra.run.vm00.stdout:163208757413 2026-03-31T21:11:52.966 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430364 got 206158430363 for osd.3 2026-03-31T21:11:52.968 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266843 got 227633266842 for osd.2 2026-03-31T21:11:52.974 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313419 got 90194313417 for osd.1 2026-03-31T21:11:52.977 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757414 got 163208757413 for osd.0 2026-03-31T21:11:53.966 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:53.969 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:53.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:53.978 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:54.211 INFO:teuthology.orchestra.run.vm00.stdout:206158430364 2026-03-31T21:11:54.224 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430364 got 206158430364 for osd.3 2026-03-31T21:11:54.224 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:54.234 INFO:teuthology.orchestra.run.vm00.stdout:227633266843 2026-03-31T21:11:54.242 INFO:teuthology.orchestra.run.vm00.stdout:90194313419 2026-03-31T21:11:54.251 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266843 got 227633266843 for osd.2 2026-03-31T21:11:54.251 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:54.259 INFO:teuthology.orchestra.run.vm00.stdout:163208757414 2026-03-31T21:11:54.261 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313419 got 90194313419 for osd.1 2026-03-31T21:11:54.261 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:54.271 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757414 got 163208757414 for osd.0 2026-03-31T21:11:54.272 DEBUG:teuthology.parallel:result is None 2026-03-31T21:11:54.272 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:11:54.516 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:11:54.517 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:11:54.528 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2026-03-31T21:11:53.873826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9036,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17820,"num_bytes_recovered":146255024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":321780,"kb_used_data":126268,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282793740,"statfs":{"total":289910292480,"available":289580789760,"internally_reserved":0,"allocated":129298432,"data_stored":128424578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":49203,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:52.703536+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:52.703536+0000","last_peered":"2026-03-31T21:11:52.703536+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:52.703536+0000","last_undegraded":"2026-03-31T21:11:52.703536+0000","last_fullsized":"2026-03-31T21:11:52.703536+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9036,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17818,"num_bytes_recovered":145926832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9036,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17818,"num_bytes_recovered":145926832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":181989376,"data_stored":181989376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430364,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757414,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47612,"kb_used_data":18092,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94324228,"statfs":{"total":96636764160,"available":96588009472,"internally_reserved":0,"allocated":18526208,"data_stored":18232534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":17743872,"data_stored":17743872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:11:54.528 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 49203, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:52.703536+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:52.703536+0000', 'last_peered': '2026-03-31T21:11:52.703536+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:52.703536+0000', 'last_undegraded': '2026-03-31T21:11:52.703536+0000', 'last_fullsized': '2026-03-31T21:11:52.703536+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9036, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17818, 'num_bytes_recovered': 145926832, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:11:54.528 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:11:59.529 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:11:59.530 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:11:59.530 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:11:59.530 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:11:59.637 INFO:teuthology.orchestra.run.vm00.stdout:163208757416 2026-03-31T21:11:59.637 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:11:59.640 INFO:teuthology.orchestra.run.vm00.stdout:206158430367 2026-03-31T21:11:59.640 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:11:59.649 INFO:teuthology.orchestra.run.vm00.stdout:90194313421 2026-03-31T21:11:59.649 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:11:59.810 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:11:59.808+0000 7f5fbaffd640 0 --2- 192.168.123.100:0/724990978 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f5fbc07f620 0x7f5fbc16fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:11:59.843 INFO:teuthology.orchestra.run.vm00.stdout:227633266845 2026-03-31T21:11:59.843 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:11:59.856 INFO:teuthology.orchestra.run.vm00.stdout:163208757415 2026-03-31T21:11:59.876 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757416 got 163208757415 for osd.0 2026-03-31T21:11:59.898 INFO:teuthology.orchestra.run.vm00.stdout:206158430365 2026-03-31T21:11:59.898 INFO:teuthology.orchestra.run.vm00.stdout:90194313420 2026-03-31T21:11:59.911 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430367 got 206158430365 for osd.3 2026-03-31T21:11:59.915 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313421 got 90194313420 for osd.1 2026-03-31T21:12:00.049 INFO:teuthology.orchestra.run.vm00.stdout:227633266844 2026-03-31T21:12:00.060 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266845 got 227633266844 for osd.2 2026-03-31T21:12:00.877 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:00.912 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:00.916 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:01.061 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:01.081 INFO:teuthology.orchestra.run.vm00.stdout:163208757417 2026-03-31T21:12:01.099 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757416 got 163208757417 for osd.0 2026-03-31T21:12:01.099 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:01.128 INFO:teuthology.orchestra.run.vm00.stdout:206158430367 2026-03-31T21:12:01.141 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430367 got 206158430367 for osd.3 2026-03-31T21:12:01.141 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:01.144 INFO:teuthology.orchestra.run.vm00.stdout:90194313421 2026-03-31T21:12:01.156 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313421 got 90194313421 for osd.1 2026-03-31T21:12:01.157 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:01.288 INFO:teuthology.orchestra.run.vm00.stdout:227633266845 2026-03-31T21:12:01.301 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266845 got 227633266845 for osd.2 2026-03-31T21:12:01.301 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:01.301 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:01.494 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:01.494 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:01.506 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2026-03-31T21:11:59.874543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8970,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17886,"num_bytes_recovered":146795696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":322048,"kb_used_data":126536,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282793472,"statfs":{"total":289910292480,"available":289580515328,"internally_reserved":0,"allocated":129572864,"data_stored":128699010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-143,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":49336,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:11:59.613835+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:11:59.613835+0000","last_peered":"2026-03-31T21:11:59.613835+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:11:59.613835+0000","last_undegraded":"2026-03-31T21:11:59.613835+0000","last_fullsized":"2026-03-31T21:11:59.613835+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8970,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17884,"num_bytes_recovered":146467504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8970,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17884,"num_bytes_recovered":146467504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":182263808,"data_stored":182263808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430367,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313421,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757417,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47880,"kb_used_data":18360,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94323960,"statfs":{"total":96636764160,"available":96587735040,"internally_reserved":0,"allocated":18800640,"data_stored":18506966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18018304,"data_stored":18018304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:01.506 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 49336, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:11:59.613835+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:11:59.613835+0000', 'last_peered': '2026-03-31T21:11:59.613835+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:11:59.613835+0000', 'last_undegraded': '2026-03-31T21:11:59.613835+0000', 'last_fullsized': '2026-03-31T21:11:59.613835+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8970, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17884, 'num_bytes_recovered': 146467504, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:01.506 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:12:06.507 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:12:06.507 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:12:06.507 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:12:06.507 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:12:06.626 INFO:teuthology.orchestra.run.vm00.stdout:163208757419 2026-03-31T21:12:06.626 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:06.630 INFO:teuthology.orchestra.run.vm00.stdout:206158430369 2026-03-31T21:12:06.630 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:06.631 INFO:teuthology.orchestra.run.vm00.stdout:227633266847 2026-03-31T21:12:06.632 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:06.637 INFO:teuthology.orchestra.run.vm00.stdout:90194313423 2026-03-31T21:12:06.637 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:06.854 INFO:teuthology.orchestra.run.vm00.stdout:227633266846 2026-03-31T21:12:06.861 INFO:teuthology.orchestra.run.vm00.stdout:206158430368 2026-03-31T21:12:06.866 INFO:teuthology.orchestra.run.vm00.stdout:90194313422 2026-03-31T21:12:06.871 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266847 got 227633266846 for osd.2 2026-03-31T21:12:06.880 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313423 got 90194313422 for osd.1 2026-03-31T21:12:06.881 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430369 got 206158430368 for osd.3 2026-03-31T21:12:06.888 INFO:teuthology.orchestra.run.vm00.stdout:163208757418 2026-03-31T21:12:06.900 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757419 got 163208757418 for osd.0 2026-03-31T21:12:07.872 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:07.881 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:07.881 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:07.900 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:08.099 INFO:teuthology.orchestra.run.vm00.stdout:227633266846 2026-03-31T21:12:08.119 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266847 got 227633266846 for osd.2 2026-03-31T21:12:08.122 INFO:teuthology.orchestra.run.vm00.stdout:163208757418 2026-03-31T21:12:08.132 INFO:teuthology.orchestra.run.vm00.stdout:206158430368 2026-03-31T21:12:08.137 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757419 got 163208757418 for osd.0 2026-03-31T21:12:08.142 INFO:teuthology.orchestra.run.vm00.stdout:90194313422 2026-03-31T21:12:08.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430369 got 206158430368 for osd.3 2026-03-31T21:12:08.155 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313423 got 90194313422 for osd.1 2026-03-31T21:12:09.119 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:09.137 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:09.147 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:09.155 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:09.374 INFO:teuthology.orchestra.run.vm00.stdout:227633266848 2026-03-31T21:12:09.380 INFO:teuthology.orchestra.run.vm00.stdout:206158430369 2026-03-31T21:12:09.383 INFO:teuthology.orchestra.run.vm00.stdout:163208757419 2026-03-31T21:12:09.385 INFO:teuthology.orchestra.run.vm00.stdout:90194313424 2026-03-31T21:12:09.393 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266847 got 227633266848 for osd.2 2026-03-31T21:12:09.393 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:09.396 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430369 got 206158430369 for osd.3 2026-03-31T21:12:09.396 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:09.397 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757419 got 163208757419 for osd.0 2026-03-31T21:12:09.397 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:09.402 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313423 got 90194313424 for osd.1 2026-03-31T21:12:09.402 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:09.402 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:09.637 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:09.637 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:09.649 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2026-03-31T21:12:07.875670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8903,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17953,"num_bytes_recovered":147344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":322320,"kb_used_data":126808,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282793200,"statfs":{"total":289910292480,"available":289580236800,"internally_reserved":0,"allocated":129851392,"data_stored":128977538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-117,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001609"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":49470,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:12:06.547545+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:12:06.547545+0000","last_peered":"2026-03-31T21:12:06.547545+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:12:06.547545+0000","last_undegraded":"2026-03-31T21:12:06.547545+0000","last_fullsized":"2026-03-31T21:12:06.547545+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8903,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17951,"num_bytes_recovered":147016368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8903,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17951,"num_bytes_recovered":147016368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":182534144,"data_stored":182534144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430369,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757419,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48152,"kb_used_data":18632,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94323688,"statfs":{"total":96636764160,"available":96587456512,"internally_reserved":0,"allocated":19079168,"data_stored":18785494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18288640,"data_stored":18288640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:09.649 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 49470, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:12:06.547545+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:12:06.547545+0000', 'last_peered': '2026-03-31T21:12:06.547545+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:12:06.547545+0000', 'last_undegraded': '2026-03-31T21:12:06.547545+0000', 'last_fullsized': '2026-03-31T21:12:06.547545+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8903, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 17951, 'num_bytes_recovered': 147016368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:09.649 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:12:14.650 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:12:14.650 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:12:14.651 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:12:14.651 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:12:14.766 INFO:teuthology.orchestra.run.vm00.stdout:227633266850 2026-03-31T21:12:14.766 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:14.773 INFO:teuthology.orchestra.run.vm00.stdout:90194313426 2026-03-31T21:12:14.773 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:14.780 INFO:teuthology.orchestra.run.vm00.stdout:163208757421 2026-03-31T21:12:14.780 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:14.787 INFO:teuthology.orchestra.run.vm00.stdout:206158430372 2026-03-31T21:12:14.787 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:14.994 INFO:teuthology.orchestra.run.vm00.stdout:227633266849 2026-03-31T21:12:15.016 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266850 got 227633266849 for osd.2 2026-03-31T21:12:15.018 INFO:teuthology.orchestra.run.vm00.stdout:90194313425 2026-03-31T21:12:15.022 INFO:teuthology.orchestra.run.vm00.stdout:206158430370 2026-03-31T21:12:15.027 INFO:teuthology.orchestra.run.vm00.stdout:163208757420 2026-03-31T21:12:15.034 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313426 got 90194313425 for osd.1 2026-03-31T21:12:15.035 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430372 got 206158430370 for osd.3 2026-03-31T21:12:15.041 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757421 got 163208757420 for osd.0 2026-03-31T21:12:16.017 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:16.035 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:16.036 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:16.042 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:16.245 INFO:teuthology.orchestra.run.vm00.stdout:227633266850 2026-03-31T21:12:16.266 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266850 got 227633266850 for osd.2 2026-03-31T21:12:16.266 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:16.285 INFO:teuthology.orchestra.run.vm00.stdout:90194313426 2026-03-31T21:12:16.286 INFO:teuthology.orchestra.run.vm00.stdout:206158430372 2026-03-31T21:12:16.299 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313426 got 90194313426 for osd.1 2026-03-31T21:12:16.299 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:16.299 INFO:teuthology.orchestra.run.vm00.stdout:163208757422 2026-03-31T21:12:16.300 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430372 got 206158430372 for osd.3 2026-03-31T21:12:16.300 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:16.312 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757421 got 163208757422 for osd.0 2026-03-31T21:12:16.312 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:16.312 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:16.565 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:16.565 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:16.576 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2026-03-31T21:12:15.876705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8823,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18033,"num_bytes_recovered":147999920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":322632,"kb_used_data":127120,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282792888,"statfs":{"total":289910292480,"available":289579917312,"internally_reserved":0,"allocated":130170880,"data_stored":129297026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001668"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":49630,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:12:14.756215+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:12:14.756215+0000","last_peered":"2026-03-31T21:12:14.756215+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:12:14.756215+0000","last_undegraded":"2026-03-31T21:12:14.756215+0000","last_fullsized":"2026-03-31T21:12:14.756215+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8823,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18031,"num_bytes_recovered":147671728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8823,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18031,"num_bytes_recovered":147671728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":182861824,"data_stored":182861824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313426,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757422,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48464,"kb_used_data":18944,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94323376,"statfs":{"total":96636764160,"available":96587137024,"internally_reserved":0,"allocated":19398656,"data_stored":19104982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18616320,"data_stored":18616320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:16.576 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 49630, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:12:14.756215+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:12:14.756215+0000', 'last_peered': '2026-03-31T21:12:14.756215+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:12:14.756215+0000', 'last_undegraded': '2026-03-31T21:12:14.756215+0000', 'last_fullsized': '2026-03-31T21:12:14.756215+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8823, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18031, 'num_bytes_recovered': 147671728, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:16.576 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:12:21.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:12:21.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:12:21.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:12:21.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:12:21.686 INFO:teuthology.orchestra.run.vm00.stdout:227633266852 2026-03-31T21:12:21.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:21.690 INFO:teuthology.orchestra.run.vm00.stdout:206158430374 2026-03-31T21:12:21.690 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:21.692 INFO:teuthology.orchestra.run.vm00.stdout:163208757424 2026-03-31T21:12:21.693 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:21.699 INFO:teuthology.orchestra.run.vm00.stdout:90194313428 2026-03-31T21:12:21.699 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:21.924 INFO:teuthology.orchestra.run.vm00.stdout:227633266851 2026-03-31T21:12:21.925 INFO:teuthology.orchestra.run.vm00.stdout:206158430373 2026-03-31T21:12:21.927 INFO:teuthology.orchestra.run.vm00.stdout:90194313427 2026-03-31T21:12:21.929 INFO:teuthology.orchestra.run.vm00.stdout:163208757423 2026-03-31T21:12:21.939 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266852 got 227633266851 for osd.2 2026-03-31T21:12:21.943 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430374 got 206158430373 for osd.3 2026-03-31T21:12:21.943 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757424 got 163208757423 for osd.0 2026-03-31T21:12:21.949 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313428 got 90194313427 for osd.1 2026-03-31T21:12:22.939 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:22.943 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:22.943 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:22.950 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:23.193 INFO:teuthology.orchestra.run.vm00.stdout:227633266853 2026-03-31T21:12:23.201 INFO:teuthology.orchestra.run.vm00.stdout:206158430374 2026-03-31T21:12:23.203 INFO:teuthology.orchestra.run.vm00.stdout:163208757424 2026-03-31T21:12:23.208 INFO:teuthology.orchestra.run.vm00.stdout:90194313428 2026-03-31T21:12:23.212 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266852 got 227633266853 for osd.2 2026-03-31T21:12:23.212 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:23.215 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430374 got 206158430374 for osd.3 2026-03-31T21:12:23.215 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:23.217 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757424 got 163208757424 for osd.0 2026-03-31T21:12:23.217 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:23.223 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313428 got 90194313428 for osd.1 2026-03-31T21:12:23.223 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:23.223 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:23.466 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:23.466 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:23.478 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2026-03-31T21:12:21.877506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8758,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18098,"num_bytes_recovered":148532400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":322904,"kb_used_data":127392,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282792616,"statfs":{"total":289910292480,"available":289579638784,"internally_reserved":0,"allocated":130449408,"data_stored":129575554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-117,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001602"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":49760,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:12:21.637514+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:12:21.637514+0000","last_peered":"2026-03-31T21:12:21.637514+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:12:21.637514+0000","last_undegraded":"2026-03-31T21:12:21.637514+0000","last_fullsized":"2026-03-31T21:12:21.637514+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8758,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18096,"num_bytes_recovered":148204208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8758,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18096,"num_bytes_recovered":148204208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183128064,"data_stored":183128064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313428,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757424,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48736,"kb_used_data":19216,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94323104,"statfs":{"total":96636764160,"available":96586858496,"internally_reserved":0,"allocated":19677184,"data_stored":19383510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18882560,"data_stored":18882560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:23.478 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 49760, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:12:21.637514+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:12:21.637514+0000', 'last_peered': '2026-03-31T21:12:21.637514+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:12:21.637514+0000', 'last_undegraded': '2026-03-31T21:12:21.637514+0000', 'last_fullsized': '2026-03-31T21:12:21.637514+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8758, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18096, 'num_bytes_recovered': 148204208, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:23.478 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:12:28.479 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:12:28.479 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:12:28.479 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:12:28.480 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:12:28.607 INFO:teuthology.orchestra.run.vm00.stdout:163208757426 2026-03-31T21:12:28.607 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:28.610 INFO:teuthology.orchestra.run.vm00.stdout:206158430376 2026-03-31T21:12:28.610 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:28.616 INFO:teuthology.orchestra.run.vm00.stdout:90194313432 2026-03-31T21:12:28.616 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:28.629 INFO:teuthology.orchestra.run.vm00.stdout:227633266855 2026-03-31T21:12:28.629 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:28.838 INFO:teuthology.orchestra.run.vm00.stdout:206158430375 2026-03-31T21:12:28.846 INFO:teuthology.orchestra.run.vm00.stdout:163208757425 2026-03-31T21:12:28.856 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430376 got 206158430375 for osd.3 2026-03-31T21:12:28.865 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757426 got 163208757425 for osd.0 2026-03-31T21:12:28.910 INFO:teuthology.orchestra.run.vm00.stdout:90194313431 2026-03-31T21:12:28.915 INFO:teuthology.orchestra.run.vm00.stdout:227633266854 2026-03-31T21:12:28.923 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313432 got 90194313431 for osd.1 2026-03-31T21:12:28.932 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266855 got 227633266854 for osd.2 2026-03-31T21:12:29.857 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:29.866 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:29.924 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:29.932 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:30.082 INFO:teuthology.orchestra.run.vm00.stdout:206158430375 2026-03-31T21:12:30.099 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430376 got 206158430375 for osd.3 2026-03-31T21:12:30.135 INFO:teuthology.orchestra.run.vm00.stdout:163208757425 2026-03-31T21:12:30.151 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757426 got 163208757425 for osd.0 2026-03-31T21:12:30.177 INFO:teuthology.orchestra.run.vm00.stdout:90194313431 2026-03-31T21:12:30.182 INFO:teuthology.orchestra.run.vm00.stdout:227633266854 2026-03-31T21:12:30.192 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313432 got 90194313431 for osd.1 2026-03-31T21:12:30.196 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266855 got 227633266854 for osd.2 2026-03-31T21:12:31.100 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:31.152 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:31.193 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:31.196 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:31.318 INFO:teuthology.orchestra.run.vm00.stdout:206158430377 2026-03-31T21:12:31.337 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430376 got 206158430377 for osd.3 2026-03-31T21:12:31.338 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:31.395 INFO:teuthology.orchestra.run.vm00.stdout:163208757427 2026-03-31T21:12:31.409 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757426 got 163208757427 for osd.0 2026-03-31T21:12:31.409 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:31.421 INFO:teuthology.orchestra.run.vm00.stdout:90194313432 2026-03-31T21:12:31.432 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313432 got 90194313432 for osd.1 2026-03-31T21:12:31.433 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:31.439 INFO:teuthology.orchestra.run.vm00.stdout:227633266855 2026-03-31T21:12:31.451 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266855 got 227633266855 for osd.2 2026-03-31T21:12:31.451 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:31.452 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:31.696 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:31.697 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:31.709 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2026-03-31T21:12:29.878581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8683,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18173,"num_bytes_recovered":149146800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":323208,"kb_used_data":127696,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282792312,"statfs":{"total":289910292480,"available":289579327488,"internally_reserved":0,"allocated":130760704,"data_stored":129886850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-140,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001616"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":49910,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:12:29.351270+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:12:29.351270+0000","last_peered":"2026-03-31T21:12:29.351270+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:12:29.351270+0000","last_undegraded":"2026-03-31T21:12:29.351270+0000","last_fullsized":"2026-03-31T21:12:29.351270+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8683,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18171,"num_bytes_recovered":148818608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8683,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18171,"num_bytes_recovered":148818608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183447552,"data_stored":183447552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430377,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313432,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757427,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49040,"kb_used_data":19520,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94322800,"statfs":{"total":96636764160,"available":96586547200,"internally_reserved":0,"allocated":19988480,"data_stored":19694806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19202048,"data_stored":19202048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:31.709 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 49910, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:12:29.351270+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:12:29.351270+0000', 'last_peered': '2026-03-31T21:12:29.351270+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:12:29.351270+0000', 'last_undegraded': '2026-03-31T21:12:29.351270+0000', 'last_fullsized': '2026-03-31T21:12:29.351270+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8683, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18171, 'num_bytes_recovered': 148818608, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:31.709 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:12:36.710 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:12:36.711 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:12:36.711 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:12:36.711 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:12:36.811 INFO:teuthology.orchestra.run.vm00.stdout:163208757429 2026-03-31T21:12:36.811 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:36.817 INFO:teuthology.orchestra.run.vm00.stdout:227633266858 2026-03-31T21:12:36.817 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:36.990 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:12:36.987+0000 7ffbda0d3640 0 --2- 192.168.123.100:0/1258122004 >> v2:192.168.123.100:3300/0 conn(0x7ffbd407f290 0x7ffbd4066fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:12:37.019 INFO:teuthology.orchestra.run.vm00.stdout:90194313434 2026-03-31T21:12:37.019 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:37.033 INFO:teuthology.orchestra.run.vm00.stdout:163208757428 2026-03-31T21:12:37.034 INFO:teuthology.orchestra.run.vm00.stdout:206158430379 2026-03-31T21:12:37.034 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:37.046 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757429 got 163208757428 for osd.0 2026-03-31T21:12:37.081 INFO:teuthology.orchestra.run.vm00.stdout:227633266856 2026-03-31T21:12:37.096 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266858 got 227633266856 for osd.2 2026-03-31T21:12:37.245 INFO:teuthology.orchestra.run.vm00.stdout:90194313433 2026-03-31T21:12:37.259 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313434 got 90194313433 for osd.1 2026-03-31T21:12:37.291 INFO:teuthology.orchestra.run.vm00.stdout:206158430378 2026-03-31T21:12:37.303 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430379 got 206158430378 for osd.3 2026-03-31T21:12:38.047 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:38.097 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:38.250 INFO:teuthology.orchestra.run.vm00.stdout:163208757429 2026-03-31T21:12:38.259 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:38.264 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757429 got 163208757429 for osd.0 2026-03-31T21:12:38.264 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:38.302 INFO:teuthology.orchestra.run.vm00.stdout:227633266858 2026-03-31T21:12:38.304 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:38.321 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266858 got 227633266858 for osd.2 2026-03-31T21:12:38.321 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:38.469 INFO:teuthology.orchestra.run.vm00.stdout:90194313435 2026-03-31T21:12:38.482 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313434 got 90194313435 for osd.1 2026-03-31T21:12:38.482 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:38.522 INFO:teuthology.orchestra.run.vm00.stdout:206158430379 2026-03-31T21:12:38.533 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430379 got 206158430379 for osd.3 2026-03-31T21:12:38.533 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:38.533 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:38.735 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:38.735 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:38.747 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2026-03-31T21:12:37.879686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8610,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18246,"num_bytes_recovered":149744816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":323476,"kb_used_data":127964,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282792044,"statfs":{"total":289910292480,"available":289579053056,"internally_reserved":0,"allocated":131035136,"data_stored":130161282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-121,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001651"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":50056,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:12:36.976600+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:12:36.976600+0000","last_peered":"2026-03-31T21:12:36.976600+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:12:36.976600+0000","last_undegraded":"2026-03-31T21:12:36.976600+0000","last_fullsized":"2026-03-31T21:12:36.976600+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8610,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18244,"num_bytes_recovered":149416624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8610,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18244,"num_bytes_recovered":149416624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183726080,"data_stored":183726080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430379,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313435,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757429,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49308,"kb_used_data":19788,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94322532,"statfs":{"total":96636764160,"available":96586272768,"internally_reserved":0,"allocated":20262912,"data_stored":19969238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19480576,"data_stored":19480576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:38.747 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 50056, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:12:36.976600+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:12:36.976600+0000', 'last_peered': '2026-03-31T21:12:36.976600+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:12:36.976600+0000', 'last_undegraded': '2026-03-31T21:12:36.976600+0000', 'last_fullsized': '2026-03-31T21:12:36.976600+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8610, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18244, 'num_bytes_recovered': 149416624, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:38.747 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:12:43.747 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:12:43.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:12:43.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:12:43.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:12:43.852 INFO:teuthology.orchestra.run.vm00.stdout:90194313437 2026-03-31T21:12:43.852 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:43.858 INFO:teuthology.orchestra.run.vm00.stdout:206158430381 2026-03-31T21:12:43.858 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:44.028 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:12:44.025+0000 7f09f6ffd640 0 --2- 192.168.123.100:0/2056245734 >> v2:192.168.123.100:3300/0 conn(0x7f09f00152c0 0x7f09f0015690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:12:44.062 INFO:teuthology.orchestra.run.vm00.stdout:163208757431 2026-03-31T21:12:44.062 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:44.070 INFO:teuthology.orchestra.run.vm00.stdout:227633266860 2026-03-31T21:12:44.070 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:44.084 INFO:teuthology.orchestra.run.vm00.stdout:90194313437 2026-03-31T21:12:44.099 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313437 got 90194313437 for osd.1 2026-03-31T21:12:44.099 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:44.135 INFO:teuthology.orchestra.run.vm00.stdout:206158430381 2026-03-31T21:12:44.147 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430381 got 206158430381 for osd.3 2026-03-31T21:12:44.147 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:44.286 INFO:teuthology.orchestra.run.vm00.stdout:163208757430 2026-03-31T21:12:44.298 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757431 got 163208757430 for osd.0 2026-03-31T21:12:44.298 INFO:teuthology.orchestra.run.vm00.stdout:227633266859 2026-03-31T21:12:44.310 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266860 got 227633266859 for osd.2 2026-03-31T21:12:45.299 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:45.311 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:45.516 INFO:teuthology.orchestra.run.vm00.stdout:163208757430 2026-03-31T21:12:45.529 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757431 got 163208757430 for osd.0 2026-03-31T21:12:45.555 INFO:teuthology.orchestra.run.vm00.stdout:227633266859 2026-03-31T21:12:45.569 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266860 got 227633266859 for osd.2 2026-03-31T21:12:46.529 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:46.570 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:46.752 INFO:teuthology.orchestra.run.vm00.stdout:163208757432 2026-03-31T21:12:46.766 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757431 got 163208757432 for osd.0 2026-03-31T21:12:46.766 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:46.785 INFO:teuthology.orchestra.run.vm00.stdout:227633266860 2026-03-31T21:12:46.799 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266860 got 227633266860 for osd.2 2026-03-31T21:12:46.799 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:46.799 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:47.053 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:47.053 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:47.065 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2026-03-31T21:12:45.880664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8539,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18317,"num_bytes_recovered":150326448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":323784,"kb_used_data":128272,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282791736,"statfs":{"total":289910292480,"available":289578737664,"internally_reserved":0,"allocated":131350528,"data_stored":130476674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001540"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":50198,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:12:44.434084+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:12:44.434084+0000","last_peered":"2026-03-31T21:12:44.434084+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:12:44.434084+0000","last_undegraded":"2026-03-31T21:12:44.434084+0000","last_fullsized":"2026-03-31T21:12:44.434084+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8539,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18315,"num_bytes_recovered":149998256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8539,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18315,"num_bytes_recovered":149998256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184037376,"data_stored":184037376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430382,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313437,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757432,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49616,"kb_used_data":20096,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94322224,"statfs":{"total":96636764160,"available":96585957376,"internally_reserved":0,"allocated":20578304,"data_stored":20284630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19791872,"data_stored":19791872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:47.066 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 50198, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:12:44.434084+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:12:44.434084+0000', 'last_peered': '2026-03-31T21:12:44.434084+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:12:44.434084+0000', 'last_undegraded': '2026-03-31T21:12:44.434084+0000', 'last_fullsized': '2026-03-31T21:12:44.434084+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8539, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18315, 'num_bytes_recovered': 149998256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:47.066 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:12:52.066 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:12:52.067 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:12:52.067 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:12:52.067 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:12:52.177 INFO:teuthology.orchestra.run.vm00.stdout:90194313439 2026-03-31T21:12:52.177 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:52.179 INFO:teuthology.orchestra.run.vm00.stdout:227633266863 2026-03-31T21:12:52.179 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:52.182 INFO:teuthology.orchestra.run.vm00.stdout:163208757434 2026-03-31T21:12:52.182 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:52.190 INFO:teuthology.orchestra.run.vm00.stdout:206158430384 2026-03-31T21:12:52.190 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:52.407 INFO:teuthology.orchestra.run.vm00.stdout:90194313438 2026-03-31T21:12:52.408 INFO:teuthology.orchestra.run.vm00.stdout:227633266862 2026-03-31T21:12:52.420 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313439 got 90194313438 for osd.1 2026-03-31T21:12:52.427 INFO:teuthology.orchestra.run.vm00.stdout:206158430383 2026-03-31T21:12:52.427 INFO:teuthology.orchestra.run.vm00.stdout:163208757433 2026-03-31T21:12:52.434 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266863 got 227633266862 for osd.2 2026-03-31T21:12:52.442 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430384 got 206158430383 for osd.3 2026-03-31T21:12:52.443 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757434 got 163208757433 for osd.0 2026-03-31T21:12:53.422 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:53.434 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:53.443 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:53.444 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:53.664 INFO:teuthology.orchestra.run.vm00.stdout:90194313438 2026-03-31T21:12:53.678 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313439 got 90194313438 for osd.1 2026-03-31T21:12:53.688 INFO:teuthology.orchestra.run.vm00.stdout:227633266862 2026-03-31T21:12:53.690 INFO:teuthology.orchestra.run.vm00.stdout:163208757433 2026-03-31T21:12:53.702 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266863 got 227633266862 for osd.2 2026-03-31T21:12:53.702 INFO:teuthology.orchestra.run.vm00.stdout:206158430383 2026-03-31T21:12:53.707 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757434 got 163208757433 for osd.0 2026-03-31T21:12:53.716 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430384 got 206158430383 for osd.3 2026-03-31T21:12:54.679 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:12:54.702 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:12:54.708 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:12:54.716 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:12:54.911 INFO:teuthology.orchestra.run.vm00.stdout:90194313440 2026-03-31T21:12:54.928 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313439 got 90194313440 for osd.1 2026-03-31T21:12:54.929 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:54.937 INFO:teuthology.orchestra.run.vm00.stdout:163208757434 2026-03-31T21:12:54.942 INFO:teuthology.orchestra.run.vm00.stdout:227633266863 2026-03-31T21:12:54.952 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757434 got 163208757434 for osd.0 2026-03-31T21:12:54.952 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:54.953 INFO:teuthology.orchestra.run.vm00.stdout:206158430384 2026-03-31T21:12:54.960 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266863 got 227633266863 for osd.2 2026-03-31T21:12:54.960 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:54.967 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430384 got 206158430384 for osd.3 2026-03-31T21:12:54.967 DEBUG:teuthology.parallel:result is None 2026-03-31T21:12:54.967 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:12:55.210 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:12:55.211 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:12:55.223 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2026-03-31T21:12:53.881759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8464,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18392,"num_bytes_recovered":150940848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":324088,"kb_used_data":128576,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282791432,"statfs":{"total":289910292480,"available":289578426368,"internally_reserved":0,"allocated":131661824,"data_stored":130787970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":50348,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:12:52.168518+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:12:52.168518+0000","last_peered":"2026-03-31T21:12:52.168518+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:12:52.168518+0000","last_undegraded":"2026-03-31T21:12:52.168518+0000","last_fullsized":"2026-03-31T21:12:52.168518+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8464,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18390,"num_bytes_recovered":150612656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8464,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18390,"num_bytes_recovered":150612656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184328192,"data_stored":184328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430384,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313440,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757434,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49920,"kb_used_data":20400,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94321920,"statfs":{"total":96636764160,"available":96585646080,"internally_reserved":0,"allocated":20889600,"data_stored":20595926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20082688,"data_stored":20082688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:12:55.224 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 50348, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:12:52.168518+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:12:52.168518+0000', 'last_peered': '2026-03-31T21:12:52.168518+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:12:52.168518+0000', 'last_undegraded': '2026-03-31T21:12:52.168518+0000', 'last_fullsized': '2026-03-31T21:12:52.168518+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8464, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18390, 'num_bytes_recovered': 150612656, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:12:55.224 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:00.224 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:00.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:00.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:00.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:00.336 INFO:teuthology.orchestra.run.vm00.stdout:90194313442 2026-03-31T21:13:00.336 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:00.342 INFO:teuthology.orchestra.run.vm00.stdout:206158430387 2026-03-31T21:13:00.343 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:00.348 INFO:teuthology.orchestra.run.vm00.stdout:227633266865 2026-03-31T21:13:00.348 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:00.354 INFO:teuthology.orchestra.run.vm00.stdout:163208757437 2026-03-31T21:13:00.354 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:00.576 INFO:teuthology.orchestra.run.vm00.stdout:206158430386 2026-03-31T21:13:00.577 INFO:teuthology.orchestra.run.vm00.stdout:227633266864 2026-03-31T21:13:00.580 INFO:teuthology.orchestra.run.vm00.stdout:90194313441 2026-03-31T21:13:00.585 INFO:teuthology.orchestra.run.vm00.stdout:163208757436 2026-03-31T21:13:00.594 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266865 got 227633266864 for osd.2 2026-03-31T21:13:00.599 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430387 got 206158430386 for osd.3 2026-03-31T21:13:00.603 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313442 got 90194313441 for osd.1 2026-03-31T21:13:00.604 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757437 got 163208757436 for osd.0 2026-03-31T21:13:01.595 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:01.599 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:01.604 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:01.605 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:01.828 INFO:teuthology.orchestra.run.vm00.stdout:227633266864 2026-03-31T21:13:01.842 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266865 got 227633266864 for osd.2 2026-03-31T21:13:01.863 INFO:teuthology.orchestra.run.vm00.stdout:163208757436 2026-03-31T21:13:01.872 INFO:teuthology.orchestra.run.vm00.stdout:90194313441 2026-03-31T21:13:01.879 INFO:teuthology.orchestra.run.vm00.stdout:206158430386 2026-03-31T21:13:01.881 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757437 got 163208757436 for osd.0 2026-03-31T21:13:01.885 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313442 got 90194313441 for osd.1 2026-03-31T21:13:01.892 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430387 got 206158430386 for osd.3 2026-03-31T21:13:02.843 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:02.882 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:02.886 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:02.892 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:03.068 INFO:teuthology.orchestra.run.vm00.stdout:227633266866 2026-03-31T21:13:03.088 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266865 got 227633266866 for osd.2 2026-03-31T21:13:03.088 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:03.117 INFO:teuthology.orchestra.run.vm00.stdout:90194313442 2026-03-31T21:13:03.118 INFO:teuthology.orchestra.run.vm00.stdout:163208757437 2026-03-31T21:13:03.129 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313442 got 90194313442 for osd.1 2026-03-31T21:13:03.129 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:03.132 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757437 got 163208757437 for osd.0 2026-03-31T21:13:03.132 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:03.161 INFO:teuthology.orchestra.run.vm00.stdout:206158430387 2026-03-31T21:13:03.173 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430387 got 206158430387 for osd.3 2026-03-31T21:13:03.173 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:03.173 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:03.422 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:03.422 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:03.434 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2026-03-31T21:13:01.882798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8385,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18471,"num_bytes_recovered":151588016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":324404,"kb_used_data":128892,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282791116,"statfs":{"total":289910292480,"available":289578102784,"internally_reserved":0,"allocated":131985408,"data_stored":131111554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-106,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":868352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":50506,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:00.314014+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:00.314014+0000","last_peered":"2026-03-31T21:13:00.314014+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:00.314014+0000","last_undegraded":"2026-03-31T21:13:00.314014+0000","last_fullsized":"2026-03-31T21:13:00.314014+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8385,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18469,"num_bytes_recovered":151259824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8385,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18469,"num_bytes_recovered":151259824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":184655872,"data_stored":184655872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430387,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313442,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757437,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50236,"kb_used_data":20716,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94321604,"statfs":{"total":96636764160,"available":96585322496,"internally_reserved":0,"allocated":21213184,"data_stored":20919510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20410368,"data_stored":20410368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:03.434 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 50506, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:00.314014+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:00.314014+0000', 'last_peered': '2026-03-31T21:13:00.314014+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:00.314014+0000', 'last_undegraded': '2026-03-31T21:13:00.314014+0000', 'last_fullsized': '2026-03-31T21:13:00.314014+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8385, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18469, 'num_bytes_recovered': 151259824, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:03.434 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:08.436 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:08.436 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:08.436 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:08.436 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:08.552 INFO:teuthology.orchestra.run.vm00.stdout:227633266868 2026-03-31T21:13:08.552 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:08.554 INFO:teuthology.orchestra.run.vm00.stdout:163208757439 2026-03-31T21:13:08.555 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:08.556 INFO:teuthology.orchestra.run.vm00.stdout:206158430389 2026-03-31T21:13:08.556 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:08.561 INFO:teuthology.orchestra.run.vm00.stdout:90194313445 2026-03-31T21:13:08.561 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:08.780 INFO:teuthology.orchestra.run.vm00.stdout:163208757438 2026-03-31T21:13:08.780 INFO:teuthology.orchestra.run.vm00.stdout:90194313444 2026-03-31T21:13:08.782 INFO:teuthology.orchestra.run.vm00.stdout:227633266867 2026-03-31T21:13:08.786 INFO:teuthology.orchestra.run.vm00.stdout:206158430388 2026-03-31T21:13:08.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313445 got 90194313444 for osd.1 2026-03-31T21:13:08.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757439 got 163208757438 for osd.0 2026-03-31T21:13:08.802 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266868 got 227633266867 for osd.2 2026-03-31T21:13:08.807 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430389 got 206158430388 for osd.3 2026-03-31T21:13:09.794 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:09.794 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:09.803 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:09.808 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:10.017 INFO:teuthology.orchestra.run.vm00.stdout:90194313444 2026-03-31T21:13:10.019 INFO:teuthology.orchestra.run.vm00.stdout:163208757438 2026-03-31T21:13:10.033 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757439 got 163208757438 for osd.0 2026-03-31T21:13:10.037 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313445 got 90194313444 for osd.1 2026-03-31T21:13:10.079 INFO:teuthology.orchestra.run.vm00.stdout:206158430388 2026-03-31T21:13:10.081 INFO:teuthology.orchestra.run.vm00.stdout:227633266867 2026-03-31T21:13:10.092 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430389 got 206158430388 for osd.3 2026-03-31T21:13:10.095 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266868 got 227633266867 for osd.2 2026-03-31T21:13:11.034 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:11.038 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:11.093 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:11.096 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:11.255 INFO:teuthology.orchestra.run.vm00.stdout:163208757440 2026-03-31T21:13:11.273 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757439 got 163208757440 for osd.0 2026-03-31T21:13:11.273 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:11.343 INFO:teuthology.orchestra.run.vm00.stdout:206158430390 2026-03-31T21:13:11.347 INFO:teuthology.orchestra.run.vm00.stdout:227633266868 2026-03-31T21:13:11.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430389 got 206158430390 for osd.3 2026-03-31T21:13:11.355 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:11.356 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:13:11.354+0000 7f44737fe640 0 --2- 192.168.123.100:0/3986892310 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f4474159200 0x7f44741795e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:13:11.363 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266868 got 227633266868 for osd.2 2026-03-31T21:13:11.363 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:11.487 INFO:teuthology.orchestra.run.vm00.stdout:90194313445 2026-03-31T21:13:11.498 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313445 got 90194313445 for osd.1 2026-03-31T21:13:11.499 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:11.499 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:11.696 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:11.696 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:11.708 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2026-03-31T21:13:09.883843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8298,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18558,"num_bytes_recovered":152300720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":324748,"kb_used_data":129236,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282790772,"statfs":{"total":289910292480,"available":289577750528,"internally_reserved":0,"allocated":132337664,"data_stored":131463810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":50680,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:09.427046+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:09.427046+0000","last_peered":"2026-03-31T21:13:09.427046+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:09.427046+0000","last_undegraded":"2026-03-31T21:13:09.427046+0000","last_fullsized":"2026-03-31T21:13:09.427046+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8298,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18556,"num_bytes_recovered":151972528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8298,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18556,"num_bytes_recovered":151972528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185024512,"data_stored":185024512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430390,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757440,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50580,"kb_used_data":21060,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94321260,"statfs":{"total":96636764160,"available":96584970240,"internally_reserved":0,"allocated":21565440,"data_stored":21271766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20779008,"data_stored":20779008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:11.709 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 50680, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:09.427046+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:09.427046+0000', 'last_peered': '2026-03-31T21:13:09.427046+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:09.427046+0000', 'last_undegraded': '2026-03-31T21:13:09.427046+0000', 'last_fullsized': '2026-03-31T21:13:09.427046+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8298, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18556, 'num_bytes_recovered': 151972528, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:11.709 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:16.710 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:16.710 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:16.710 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:16.710 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:16.831 INFO:teuthology.orchestra.run.vm00.stdout:163208757442 2026-03-31T21:13:16.831 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:16.834 INFO:teuthology.orchestra.run.vm00.stdout:206158430392 2026-03-31T21:13:16.834 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:16.835 INFO:teuthology.orchestra.run.vm00.stdout:227633266871 2026-03-31T21:13:16.835 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:16.842 INFO:teuthology.orchestra.run.vm00.stdout:90194313447 2026-03-31T21:13:16.842 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:17.057 INFO:teuthology.orchestra.run.vm00.stdout:206158430391 2026-03-31T21:13:17.068 INFO:teuthology.orchestra.run.vm00.stdout:90194313446 2026-03-31T21:13:17.068 INFO:teuthology.orchestra.run.vm00.stdout:227633266869 2026-03-31T21:13:17.080 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430392 got 206158430391 for osd.3 2026-03-31T21:13:17.083 INFO:teuthology.orchestra.run.vm00.stdout:163208757441 2026-03-31T21:13:17.088 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313447 got 90194313446 for osd.1 2026-03-31T21:13:17.089 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266871 got 227633266869 for osd.2 2026-03-31T21:13:17.099 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757442 got 163208757441 for osd.0 2026-03-31T21:13:18.081 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:18.088 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:18.090 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:18.100 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:18.305 INFO:teuthology.orchestra.run.vm00.stdout:206158430392 2026-03-31T21:13:18.320 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430392 got 206158430392 for osd.3 2026-03-31T21:13:18.320 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:18.351 INFO:teuthology.orchestra.run.vm00.stdout:163208757442 2026-03-31T21:13:18.359 INFO:teuthology.orchestra.run.vm00.stdout:227633266871 2026-03-31T21:13:18.363 INFO:teuthology.orchestra.run.vm00.stdout:90194313448 2026-03-31T21:13:18.363 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757442 got 163208757442 for osd.0 2026-03-31T21:13:18.364 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:18.374 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266871 got 227633266871 for osd.2 2026-03-31T21:13:18.374 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:18.376 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313447 got 90194313448 for osd.1 2026-03-31T21:13:18.376 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:18.376 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:18.630 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:18.631 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:18.643 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2026-03-31T21:13:17.885084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8227,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18629,"num_bytes_recovered":152882352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":325016,"kb_used_data":129504,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282790504,"statfs":{"total":289910292480,"available":289577476096,"internally_reserved":0,"allocated":132612096,"data_stored":131738242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-118,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001744"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":50822,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:16.786662+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:16.786662+0000","last_peered":"2026-03-31T21:13:16.786662+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:16.786662+0000","last_undegraded":"2026-03-31T21:13:16.786662+0000","last_fullsized":"2026-03-31T21:13:16.786662+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8227,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18627,"num_bytes_recovered":152554160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8227,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18627,"num_bytes_recovered":152554160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185303040,"data_stored":185303040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430392,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757442,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50848,"kb_used_data":21328,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94320992,"statfs":{"total":96636764160,"available":96584695808,"internally_reserved":0,"allocated":21839872,"data_stored":21546198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21057536,"data_stored":21057536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:18.643 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 50822, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:16.786662+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:16.786662+0000', 'last_peered': '2026-03-31T21:13:16.786662+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:16.786662+0000', 'last_undegraded': '2026-03-31T21:13:16.786662+0000', 'last_fullsized': '2026-03-31T21:13:16.786662+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8227, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18627, 'num_bytes_recovered': 152554160, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:18.643 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:23.643 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:23.644 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:23.644 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:23.644 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:23.758 INFO:teuthology.orchestra.run.vm00.stdout:90194313450 2026-03-31T21:13:23.758 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:23.761 INFO:teuthology.orchestra.run.vm00.stdout:206158430394 2026-03-31T21:13:23.762 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:23.766 INFO:teuthology.orchestra.run.vm00.stdout:227633266873 2026-03-31T21:13:23.766 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:23.771 INFO:teuthology.orchestra.run.vm00.stdout:163208757444 2026-03-31T21:13:23.771 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:24.000 INFO:teuthology.orchestra.run.vm00.stdout:227633266872 2026-03-31T21:13:24.000 INFO:teuthology.orchestra.run.vm00.stdout:206158430393 2026-03-31T21:13:24.004 INFO:teuthology.orchestra.run.vm00.stdout:163208757443 2026-03-31T21:13:24.006 INFO:teuthology.orchestra.run.vm00.stdout:90194313448 2026-03-31T21:13:24.014 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430394 got 206158430393 for osd.3 2026-03-31T21:13:24.016 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266873 got 227633266872 for osd.2 2026-03-31T21:13:24.017 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757444 got 163208757443 for osd.0 2026-03-31T21:13:24.025 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313450 got 90194313448 for osd.1 2026-03-31T21:13:25.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:25.017 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:25.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:25.025 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:25.242 INFO:teuthology.orchestra.run.vm00.stdout:227633266873 2026-03-31T21:13:25.252 INFO:teuthology.orchestra.run.vm00.stdout:206158430394 2026-03-31T21:13:25.262 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266873 got 227633266873 for osd.2 2026-03-31T21:13:25.262 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:25.270 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430394 got 206158430394 for osd.3 2026-03-31T21:13:25.270 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:25.286 INFO:teuthology.orchestra.run.vm00.stdout:163208757444 2026-03-31T21:13:25.286 INFO:teuthology.orchestra.run.vm00.stdout:90194313450 2026-03-31T21:13:25.299 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757444 got 163208757444 for osd.0 2026-03-31T21:13:25.299 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:25.301 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313450 got 90194313450 for osd.1 2026-03-31T21:13:25.301 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:25.301 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:25.534 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:25.534 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:25.546 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2026-03-31T21:13:23.885905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8160,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18696,"num_bytes_recovered":153431216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":325292,"kb_used_data":129780,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282790228,"statfs":{"total":289910292480,"available":289577193472,"internally_reserved":0,"allocated":132894720,"data_stored":132020866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-138,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001679"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":50956,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:23.675944+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:23.675944+0000","last_peered":"2026-03-31T21:13:23.675944+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:23.675944+0000","last_undegraded":"2026-03-31T21:13:23.675944+0000","last_fullsized":"2026-03-31T21:13:23.675944+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8160,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18694,"num_bytes_recovered":153103024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8160,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18694,"num_bytes_recovered":153103024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185577472,"data_stored":185577472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430394,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313450,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51124,"kb_used_data":21604,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94320716,"statfs":{"total":96636764160,"available":96584413184,"internally_reserved":0,"allocated":22122496,"data_stored":21828822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21331968,"data_stored":21331968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:25.546 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 50956, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:23.675944+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:23.675944+0000', 'last_peered': '2026-03-31T21:13:23.675944+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:23.675944+0000', 'last_undegraded': '2026-03-31T21:13:23.675944+0000', 'last_fullsized': '2026-03-31T21:13:23.675944+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8160, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18694, 'num_bytes_recovered': 153103024, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:25.547 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:30.547 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:30.547 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:30.547 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:30.547 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:30.652 INFO:teuthology.orchestra.run.vm00.stdout:206158430397 2026-03-31T21:13:30.652 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:30.654 INFO:teuthology.orchestra.run.vm00.stdout:227633266875 2026-03-31T21:13:30.654 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:30.658 INFO:teuthology.orchestra.run.vm00.stdout:90194313452 2026-03-31T21:13:30.658 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:30.660 INFO:teuthology.orchestra.run.vm00.stdout:163208757447 2026-03-31T21:13:30.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:30.881 INFO:teuthology.orchestra.run.vm00.stdout:90194313451 2026-03-31T21:13:30.888 INFO:teuthology.orchestra.run.vm00.stdout:206158430396 2026-03-31T21:13:30.888 INFO:teuthology.orchestra.run.vm00.stdout:227633266874 2026-03-31T21:13:30.898 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313452 got 90194313451 for osd.1 2026-03-31T21:13:30.900 INFO:teuthology.orchestra.run.vm00.stdout:163208757446 2026-03-31T21:13:30.903 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266875 got 227633266874 for osd.2 2026-03-31T21:13:30.908 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430397 got 206158430396 for osd.3 2026-03-31T21:13:30.913 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757447 got 163208757446 for osd.0 2026-03-31T21:13:31.899 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:31.904 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:31.909 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:31.914 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:32.138 INFO:teuthology.orchestra.run.vm00.stdout:90194313452 2026-03-31T21:13:32.152 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313452 got 90194313452 for osd.1 2026-03-31T21:13:32.152 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:32.165 INFO:teuthology.orchestra.run.vm00.stdout:206158430397 2026-03-31T21:13:32.172 INFO:teuthology.orchestra.run.vm00.stdout:227633266876 2026-03-31T21:13:32.182 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430397 got 206158430397 for osd.3 2026-03-31T21:13:32.182 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:32.185 INFO:teuthology.orchestra.run.vm00.stdout:163208757447 2026-03-31T21:13:32.185 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266875 got 227633266876 for osd.2 2026-03-31T21:13:32.185 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:32.196 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757447 got 163208757447 for osd.0 2026-03-31T21:13:32.196 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:32.197 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:32.433 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:32.433 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:32.445 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2026-03-31T21:13:31.887062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8094,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18762,"num_bytes_recovered":153971888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":325564,"kb_used_data":130052,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282789956,"statfs":{"total":289910292480,"available":289576914944,"internally_reserved":0,"allocated":133173248,"data_stored":132299394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001627"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":51088,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:30.564721+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:30.564721+0000","last_peered":"2026-03-31T21:13:30.564721+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:30.564721+0000","last_undegraded":"2026-03-31T21:13:30.564721+0000","last_fullsized":"2026-03-31T21:13:30.564721+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18760,"num_bytes_recovered":153643696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":53,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:09:37.558979+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:09:37.558979+0000","last_peered":"2026-03-31T21:09:37.558979+0000","last_clean":"2026-03-31T21:09:37.558979+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:09:37.558979+0000","last_undegraded":"2026-03-31T21:09:37.558979+0000","last_fullsized":"2026-03-31T21:09:37.558979+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18760,"num_bytes_recovered":153643696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":185847808,"data_stored":185847808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430397,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266876,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313452,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51396,"kb_used_data":21876,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94320444,"statfs":{"total":96636764160,"available":96584134656,"internally_reserved":0,"allocated":22401024,"data_stored":22107350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21602304,"data_stored":21602304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:32.445 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 51088, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:30.564721+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:30.564721+0000', 'last_peered': '2026-03-31T21:13:30.564721+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:30.564721+0000', 'last_undegraded': '2026-03-31T21:13:30.564721+0000', 'last_fullsized': '2026-03-31T21:13:30.564721+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8094, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18760, 'num_bytes_recovered': 153643696, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:32.445 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:37.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:37.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:37.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:37.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:37.561 INFO:teuthology.orchestra.run.vm00.stdout:90194313454 2026-03-31T21:13:37.561 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:37.562 INFO:teuthology.orchestra.run.vm00.stdout:227633266878 2026-03-31T21:13:37.563 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:37.566 INFO:teuthology.orchestra.run.vm00.stdout:163208757449 2026-03-31T21:13:37.567 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:37.572 INFO:teuthology.orchestra.run.vm00.stdout:206158430399 2026-03-31T21:13:37.572 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:37.795 INFO:teuthology.orchestra.run.vm00.stdout:227633266876 2026-03-31T21:13:37.801 INFO:teuthology.orchestra.run.vm00.stdout:90194313453 2026-03-31T21:13:37.805 INFO:teuthology.orchestra.run.vm00.stdout:206158430398 2026-03-31T21:13:37.810 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266878 got 227633266876 for osd.2 2026-03-31T21:13:37.818 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313454 got 90194313453 for osd.1 2026-03-31T21:13:37.819 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430399 got 206158430398 for osd.3 2026-03-31T21:13:37.854 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:13:37.852+0000 7f84bdd74640 0 --2- 192.168.123.100:0/3031693936 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f84c007f830 0x7f84c0076ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:13:37.982 INFO:teuthology.orchestra.run.vm00.stdout:163208757448 2026-03-31T21:13:37.995 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757449 got 163208757448 for osd.0 2026-03-31T21:13:38.811 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:38.819 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:38.819 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:38.996 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:39.019 INFO:teuthology.orchestra.run.vm00.stdout:227633266878 2026-03-31T21:13:39.034 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266878 got 227633266878 for osd.2 2026-03-31T21:13:39.035 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:39.069 INFO:teuthology.orchestra.run.vm00.stdout:90194313455 2026-03-31T21:13:39.070 INFO:teuthology.orchestra.run.vm00.stdout:206158430399 2026-03-31T21:13:39.084 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313454 got 90194313455 for osd.1 2026-03-31T21:13:39.084 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:39.088 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430399 got 206158430399 for osd.3 2026-03-31T21:13:39.088 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:39.221 INFO:teuthology.orchestra.run.vm00.stdout:163208757449 2026-03-31T21:13:39.232 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757449 got 163208757449 for osd.0 2026-03-31T21:13:39.232 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:39.233 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:39.433 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:39.433 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:39.445 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":432,"stamp":"2026-03-31T21:13:37.887803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8027,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18829,"num_bytes_recovered":154520752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":325832,"kb_used_data":130320,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282789688,"statfs":{"total":289910292480,"available":289576640512,"internally_reserved":0,"allocated":133447680,"data_stored":132573826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":51222,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:37.458802+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:37.458802+0000","last_peered":"2026-03-31T21:13:37.458802+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:37.458802+0000","last_undegraded":"2026-03-31T21:13:37.458802+0000","last_fullsized":"2026-03-31T21:13:37.458802+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8027,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18827,"num_bytes_recovered":154192560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8027,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18827,"num_bytes_recovered":154192560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186122240,"data_stored":186122240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430399,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266878,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313455,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757449,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51664,"kb_used_data":22144,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94320176,"statfs":{"total":96636764160,"available":96583860224,"internally_reserved":0,"allocated":22675456,"data_stored":22381782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21876736,"data_stored":21876736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:39.445 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 51222, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:37.458802+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:37.458802+0000', 'last_peered': '2026-03-31T21:13:37.458802+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:37.458802+0000', 'last_undegraded': '2026-03-31T21:13:37.458802+0000', 'last_fullsized': '2026-03-31T21:13:37.458802+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8027, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18827, 'num_bytes_recovered': 154192560, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:39.445 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:44.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:44.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:44.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:44.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:44.554 INFO:teuthology.orchestra.run.vm00.stdout:206158430402 2026-03-31T21:13:44.554 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:44.555 INFO:teuthology.orchestra.run.vm00.stdout:163208757451 2026-03-31T21:13:44.555 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:44.561 INFO:teuthology.orchestra.run.vm00.stdout:227633266880 2026-03-31T21:13:44.561 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:44.762 INFO:teuthology.orchestra.run.vm00.stdout:163208757450 2026-03-31T21:13:44.763 INFO:teuthology.orchestra.run.vm00.stdout:90194313457 2026-03-31T21:13:44.763 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:44.763 INFO:teuthology.orchestra.run.vm00.stdout:227633266879 2026-03-31T21:13:44.770 INFO:teuthology.orchestra.run.vm00.stdout:206158430400 2026-03-31T21:13:44.777 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266880 got 227633266879 for osd.2 2026-03-31T21:13:44.781 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757451 got 163208757450 for osd.0 2026-03-31T21:13:44.787 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430402 got 206158430400 for osd.3 2026-03-31T21:13:44.962 INFO:teuthology.orchestra.run.vm00.stdout:90194313456 2026-03-31T21:13:44.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313457 got 90194313456 for osd.1 2026-03-31T21:13:45.778 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:45.782 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:45.787 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:45.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:45.984 INFO:teuthology.orchestra.run.vm00.stdout:227633266879 2026-03-31T21:13:45.999 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266880 got 227633266879 for osd.2 2026-03-31T21:13:46.027 INFO:teuthology.orchestra.run.vm00.stdout:163208757450 2026-03-31T21:13:46.033 INFO:teuthology.orchestra.run.vm00.stdout:206158430400 2026-03-31T21:13:46.047 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757451 got 163208757450 for osd.0 2026-03-31T21:13:46.060 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430402 got 206158430400 for osd.3 2026-03-31T21:13:46.174 INFO:teuthology.orchestra.run.vm00.stdout:90194313457 2026-03-31T21:13:46.185 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313457 got 90194313457 for osd.1 2026-03-31T21:13:46.185 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:47.000 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:47.048 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:47.060 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:47.225 INFO:teuthology.orchestra.run.vm00.stdout:227633266880 2026-03-31T21:13:47.237 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266880 got 227633266880 for osd.2 2026-03-31T21:13:47.237 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:47.266 INFO:teuthology.orchestra.run.vm00.stdout:163208757452 2026-03-31T21:13:47.279 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757451 got 163208757452 for osd.0 2026-03-31T21:13:47.279 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:47.294 INFO:teuthology.orchestra.run.vm00.stdout:206158430402 2026-03-31T21:13:47.305 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430402 got 206158430402 for osd.3 2026-03-31T21:13:47.305 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:47.306 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:47.545 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:47.545 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:47.556 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2026-03-31T21:13:45.888772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7959,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18897,"num_bytes_recovered":155077808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":326100,"kb_used_data":130588,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282789420,"statfs":{"total":289910292480,"available":289576366080,"internally_reserved":0,"allocated":133722112,"data_stored":132848258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-135,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001450"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":51358,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:44.530120+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:44.530120+0000","last_peered":"2026-03-31T21:13:44.530120+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:44.530120+0000","last_undegraded":"2026-03-31T21:13:44.530120+0000","last_fullsized":"2026-03-31T21:13:44.530120+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7959,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18895,"num_bytes_recovered":154749616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7959,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18895,"num_bytes_recovered":154749616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186408960,"data_stored":186408960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430402,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266880,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757452,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51932,"kb_used_data":22412,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94319908,"statfs":{"total":96636764160,"available":96583585792,"internally_reserved":0,"allocated":22949888,"data_stored":22656214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22163456,"data_stored":22163456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:47.557 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 51358, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:44.530120+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:44.530120+0000', 'last_peered': '2026-03-31T21:13:44.530120+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:44.530120+0000', 'last_undegraded': '2026-03-31T21:13:44.530120+0000', 'last_fullsized': '2026-03-31T21:13:44.530120+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7959, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18895, 'num_bytes_recovered': 154749616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:47.557 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:52.557 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:52.557 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:52.557 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:52.558 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:52.665 INFO:teuthology.orchestra.run.vm00.stdout:206158430404 2026-03-31T21:13:52.665 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:52.671 INFO:teuthology.orchestra.run.vm00.stdout:227633266883 2026-03-31T21:13:52.671 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:52.673 INFO:teuthology.orchestra.run.vm00.stdout:90194313460 2026-03-31T21:13:52.673 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:52.674 INFO:teuthology.orchestra.run.vm00.stdout:163208757454 2026-03-31T21:13:52.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:52.899 INFO:teuthology.orchestra.run.vm00.stdout:227633266882 2026-03-31T21:13:52.910 INFO:teuthology.orchestra.run.vm00.stdout:163208757453 2026-03-31T21:13:52.913 INFO:teuthology.orchestra.run.vm00.stdout:90194313458 2026-03-31T21:13:52.918 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266883 got 227633266882 for osd.2 2026-03-31T21:13:52.922 INFO:teuthology.orchestra.run.vm00.stdout:206158430403 2026-03-31T21:13:52.927 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757454 got 163208757453 for osd.0 2026-03-31T21:13:52.928 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313460 got 90194313458 for osd.1 2026-03-31T21:13:52.933 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430404 got 206158430403 for osd.3 2026-03-31T21:13:53.919 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:53.928 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:53.929 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:53.934 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:54.179 INFO:teuthology.orchestra.run.vm00.stdout:227633266883 2026-03-31T21:13:54.197 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266883 got 227633266883 for osd.2 2026-03-31T21:13:54.197 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:54.204 INFO:teuthology.orchestra.run.vm00.stdout:163208757454 2026-03-31T21:13:54.207 INFO:teuthology.orchestra.run.vm00.stdout:90194313460 2026-03-31T21:13:54.217 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757454 got 163208757454 for osd.0 2026-03-31T21:13:54.217 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:54.218 INFO:teuthology.orchestra.run.vm00.stdout:206158430404 2026-03-31T21:13:54.226 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313460 got 90194313460 for osd.1 2026-03-31T21:13:54.226 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:54.230 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430404 got 206158430404 for osd.3 2026-03-31T21:13:54.231 DEBUG:teuthology.parallel:result is None 2026-03-31T21:13:54.231 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:13:54.465 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:13:54.466 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:13:54.476 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2026-03-31T21:13:53.889710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7880,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18976,"num_bytes_recovered":155724976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":326408,"kb_used_data":130896,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282789112,"statfs":{"total":289910292480,"available":289576050688,"internally_reserved":0,"allocated":134037504,"data_stored":133163650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-128,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":1048576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001396"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":51516,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:52.604454+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:52.604454+0000","last_peered":"2026-03-31T21:13:52.604454+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:52.604454+0000","last_undegraded":"2026-03-31T21:13:52.604454+0000","last_fullsized":"2026-03-31T21:13:52.604454+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7880,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18974,"num_bytes_recovered":155396784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7880,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18974,"num_bytes_recovered":155396784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":186724352,"data_stored":186724352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430404,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313460,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757454,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52240,"kb_used_data":22720,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94319600,"statfs":{"total":96636764160,"available":96583270400,"internally_reserved":0,"allocated":23265280,"data_stored":22971606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22478848,"data_stored":22478848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:13:54.477 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 51516, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:52.604454+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:52.604454+0000', 'last_peered': '2026-03-31T21:13:52.604454+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:52.604454+0000', 'last_undegraded': '2026-03-31T21:13:52.604454+0000', 'last_fullsized': '2026-03-31T21:13:52.604454+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7880, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18974, 'num_bytes_recovered': 155396784, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:13:54.477 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:13:59.478 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:13:59.478 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:13:59.478 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:13:59.478 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:13:59.596 INFO:teuthology.orchestra.run.vm00.stdout:90194313462 2026-03-31T21:13:59.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:13:59.596 INFO:teuthology.orchestra.run.vm00.stdout:163208757456 2026-03-31T21:13:59.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:13:59.599 INFO:teuthology.orchestra.run.vm00.stdout:227633266885 2026-03-31T21:13:59.599 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:13:59.606 INFO:teuthology.orchestra.run.vm00.stdout:206158430407 2026-03-31T21:13:59.606 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:13:59.815 INFO:teuthology.orchestra.run.vm00.stdout:90194313461 2026-03-31T21:13:59.832 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313462 got 90194313461 for osd.1 2026-03-31T21:13:59.837 INFO:teuthology.orchestra.run.vm00.stdout:163208757455 2026-03-31T21:13:59.850 INFO:teuthology.orchestra.run.vm00.stdout:227633266884 2026-03-31T21:13:59.856 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757456 got 163208757455 for osd.0 2026-03-31T21:13:59.866 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266885 got 227633266884 for osd.2 2026-03-31T21:13:59.904 INFO:teuthology.orchestra.run.vm00.stdout:206158430405 2026-03-31T21:13:59.917 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430407 got 206158430405 for osd.3 2026-03-31T21:14:00.833 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:00.856 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:00.866 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:00.917 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:01.053 INFO:teuthology.orchestra.run.vm00.stdout:90194313462 2026-03-31T21:14:01.070 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313462 got 90194313462 for osd.1 2026-03-31T21:14:01.070 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:01.096 INFO:teuthology.orchestra.run.vm00.stdout:227633266885 2026-03-31T21:14:01.106 INFO:teuthology.orchestra.run.vm00.stdout:163208757457 2026-03-31T21:14:01.109 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266885 got 227633266885 for osd.2 2026-03-31T21:14:01.109 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:01.119 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757456 got 163208757457 for osd.0 2026-03-31T21:14:01.119 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:01.174 INFO:teuthology.orchestra.run.vm00.stdout:206158430407 2026-03-31T21:14:01.187 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430407 got 206158430407 for osd.3 2026-03-31T21:14:01.187 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:01.187 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:01.382 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:01.382 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:01.394 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2026-03-31T21:13:59.890514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7813,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19043,"num_bytes_recovered":156273840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":326684,"kb_used_data":131172,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282788836,"statfs":{"total":289910292480,"available":289575768064,"internally_reserved":0,"allocated":134320128,"data_stored":133446274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":51651,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:13:59.491083+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:13:59.491083+0000","last_peered":"2026-03-31T21:13:59.491083+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:13:59.491083+0000","last_undegraded":"2026-03-31T21:13:59.491083+0000","last_fullsized":"2026-03-31T21:13:59.491083+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7813,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19041,"num_bytes_recovered":155945648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7813,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19041,"num_bytes_recovered":155945648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187006976,"data_stored":187006976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430407,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313462,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757457,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52516,"kb_used_data":22996,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94319324,"statfs":{"total":96636764160,"available":96582987776,"internally_reserved":0,"allocated":23547904,"data_stored":23254230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22761472,"data_stored":22761472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:01.394 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 51651, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:13:59.491083+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:13:59.491083+0000', 'last_peered': '2026-03-31T21:13:59.491083+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:13:59.491083+0000', 'last_undegraded': '2026-03-31T21:13:59.491083+0000', 'last_fullsized': '2026-03-31T21:13:59.491083+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7813, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19041, 'num_bytes_recovered': 155945648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:01.394 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:06.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:06.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:06.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:06.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:06.504 INFO:teuthology.orchestra.run.vm00.stdout:163208757459 2026-03-31T21:14:06.504 INFO:teuthology.orchestra.run.vm00.stdout:227633266887 2026-03-31T21:14:06.505 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:06.505 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:06.511 INFO:teuthology.orchestra.run.vm00.stdout:206158430409 2026-03-31T21:14:06.512 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:06.519 INFO:teuthology.orchestra.run.vm00.stdout:90194313464 2026-03-31T21:14:06.519 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:06.748 INFO:teuthology.orchestra.run.vm00.stdout:163208757458 2026-03-31T21:14:06.749 INFO:teuthology.orchestra.run.vm00.stdout:227633266886 2026-03-31T21:14:06.761 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266887 got 227633266886 for osd.2 2026-03-31T21:14:06.764 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757459 got 163208757458 for osd.0 2026-03-31T21:14:06.783 INFO:teuthology.orchestra.run.vm00.stdout:90194313463 2026-03-31T21:14:06.795 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313464 got 90194313463 for osd.1 2026-03-31T21:14:06.865 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:14:06.862+0000 7f323a1b7640 0 --2- 192.168.123.100:0/92631236 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f3234133ce0 0x7f32341340b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:14:06.998 INFO:teuthology.orchestra.run.vm00.stdout:206158430408 2026-03-31T21:14:07.010 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430409 got 206158430408 for osd.3 2026-03-31T21:14:07.762 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:07.765 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:07.796 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:07.968 INFO:teuthology.orchestra.run.vm00.stdout:227633266886 2026-03-31T21:14:07.980 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266887 got 227633266886 for osd.2 2026-03-31T21:14:08.012 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:08.014 INFO:teuthology.orchestra.run.vm00.stdout:163208757458 2026-03-31T21:14:08.022 INFO:teuthology.orchestra.run.vm00.stdout:90194313463 2026-03-31T21:14:08.030 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757459 got 163208757458 for osd.0 2026-03-31T21:14:08.038 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313464 got 90194313463 for osd.1 2026-03-31T21:14:08.223 INFO:teuthology.orchestra.run.vm00.stdout:206158430409 2026-03-31T21:14:08.235 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430409 got 206158430409 for osd.3 2026-03-31T21:14:08.235 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:08.981 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:09.031 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:09.039 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:09.195 INFO:teuthology.orchestra.run.vm00.stdout:227633266888 2026-03-31T21:14:09.210 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266887 got 227633266888 for osd.2 2026-03-31T21:14:09.210 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:09.249 INFO:teuthology.orchestra.run.vm00.stdout:163208757459 2026-03-31T21:14:09.261 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757459 got 163208757459 for osd.0 2026-03-31T21:14:09.261 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:09.272 INFO:teuthology.orchestra.run.vm00.stdout:90194313465 2026-03-31T21:14:09.283 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313464 got 90194313465 for osd.1 2026-03-31T21:14:09.284 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:09.284 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:09.514 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:09.514 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:09.525 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2026-03-31T21:14:07.893717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7746,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19110,"num_bytes_recovered":156822704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":326952,"kb_used_data":131440,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282788568,"statfs":{"total":289910292480,"available":289575493632,"internally_reserved":0,"allocated":134594560,"data_stored":133720706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-117,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003729"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":51785,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:06.458384+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:06.458384+0000","last_peered":"2026-03-31T21:14:06.458384+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:06.458384+0000","last_undegraded":"2026-03-31T21:14:06.458384+0000","last_fullsized":"2026-03-31T21:14:06.458384+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7746,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19108,"num_bytes_recovered":156494512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7746,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19108,"num_bytes_recovered":156494512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187273216,"data_stored":187273216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430409,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266888,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313465,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757459,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52784,"kb_used_data":23264,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94319056,"statfs":{"total":96636764160,"available":96582713344,"internally_reserved":0,"allocated":23822336,"data_stored":23528662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23027712,"data_stored":23027712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:09.525 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 51785, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:06.458384+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:06.458384+0000', 'last_peered': '2026-03-31T21:14:06.458384+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:06.458384+0000', 'last_undegraded': '2026-03-31T21:14:06.458384+0000', 'last_fullsized': '2026-03-31T21:14:06.458384+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7746, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19108, 'num_bytes_recovered': 156494512, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:09.525 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:14.526 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:14.526 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:14.526 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:14.526 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:14.632 INFO:teuthology.orchestra.run.vm00.stdout:227633266890 2026-03-31T21:14:14.633 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:14.634 INFO:teuthology.orchestra.run.vm00.stdout:90194313467 2026-03-31T21:14:14.634 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:14.644 INFO:teuthology.orchestra.run.vm00.stdout:206158430412 2026-03-31T21:14:14.644 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:14.838 INFO:teuthology.orchestra.run.vm00.stdout:163208757462 2026-03-31T21:14:14.838 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:14.849 INFO:teuthology.orchestra.run.vm00.stdout:90194313466 2026-03-31T21:14:14.852 INFO:teuthology.orchestra.run.vm00.stdout:227633266889 2026-03-31T21:14:14.860 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313467 got 90194313466 for osd.1 2026-03-31T21:14:14.864 INFO:teuthology.orchestra.run.vm00.stdout:206158430410 2026-03-31T21:14:14.865 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266890 got 227633266889 for osd.2 2026-03-31T21:14:14.875 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430412 got 206158430410 for osd.3 2026-03-31T21:14:15.046 INFO:teuthology.orchestra.run.vm00.stdout:163208757460 2026-03-31T21:14:15.057 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757462 got 163208757460 for osd.0 2026-03-31T21:14:15.860 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:15.865 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:15.875 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:16.057 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:16.071 INFO:teuthology.orchestra.run.vm00.stdout:90194313466 2026-03-31T21:14:16.085 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313467 got 90194313466 for osd.1 2026-03-31T21:14:16.120 INFO:teuthology.orchestra.run.vm00.stdout:227633266889 2026-03-31T21:14:16.122 INFO:teuthology.orchestra.run.vm00.stdout:206158430410 2026-03-31T21:14:16.133 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266890 got 227633266889 for osd.2 2026-03-31T21:14:16.134 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430412 got 206158430410 for osd.3 2026-03-31T21:14:16.270 INFO:teuthology.orchestra.run.vm00.stdout:163208757462 2026-03-31T21:14:16.282 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757462 got 163208757462 for osd.0 2026-03-31T21:14:16.282 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:17.086 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:17.135 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:17.135 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:17.294 INFO:teuthology.orchestra.run.vm00.stdout:90194313467 2026-03-31T21:14:17.306 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313467 got 90194313467 for osd.1 2026-03-31T21:14:17.306 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:17.347 INFO:teuthology.orchestra.run.vm00.stdout:206158430412 2026-03-31T21:14:17.359 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430412 got 206158430412 for osd.3 2026-03-31T21:14:17.359 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:17.420 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:14:17.417+0000 7ff5a9a47640 0 --2- 192.168.123.100:0/624068951 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7ff5a41758e0 0x7ff5a4076ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:14:17.547 INFO:teuthology.orchestra.run.vm00.stdout:227633266890 2026-03-31T21:14:17.559 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266890 got 227633266890 for osd.2 2026-03-31T21:14:17.559 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:17.559 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:17.759 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:17.760 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:17.772 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2026-03-31T21:14:15.894746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7669,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19187,"num_bytes_recovered":157453488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":327256,"kb_used_data":131744,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282788264,"statfs":{"total":289910292480,"available":289575182336,"internally_reserved":0,"allocated":134905856,"data_stored":134032002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003710"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":51939,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:14.566339+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:14.566339+0000","last_peered":"2026-03-31T21:14:14.566339+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:14.566339+0000","last_undegraded":"2026-03-31T21:14:14.566339+0000","last_fullsized":"2026-03-31T21:14:14.566339+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7669,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19185,"num_bytes_recovered":157125296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7669,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19185,"num_bytes_recovered":157125296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187596800,"data_stored":187596800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430412,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266890,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757462,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53088,"kb_used_data":23568,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94318752,"statfs":{"total":96636764160,"available":96582402048,"internally_reserved":0,"allocated":24133632,"data_stored":23839958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23351296,"data_stored":23351296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:17.772 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 51939, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:14.566339+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:14.566339+0000', 'last_peered': '2026-03-31T21:14:14.566339+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:14.566339+0000', 'last_undegraded': '2026-03-31T21:14:14.566339+0000', 'last_fullsized': '2026-03-31T21:14:14.566339+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7669, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19185, 'num_bytes_recovered': 157125296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:17.772 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:22.773 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:22.774 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:22.774 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:22.774 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:22.876 INFO:teuthology.orchestra.run.vm00.stdout:227633266893 2026-03-31T21:14:22.877 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:22.879 INFO:teuthology.orchestra.run.vm00.stdout:163208757464 2026-03-31T21:14:22.879 INFO:teuthology.orchestra.run.vm00.stdout:206158430414 2026-03-31T21:14:22.879 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:22.879 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:22.884 INFO:teuthology.orchestra.run.vm00.stdout:90194313470 2026-03-31T21:14:22.884 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:23.106 INFO:teuthology.orchestra.run.vm00.stdout:227633266892 2026-03-31T21:14:23.124 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266893 got 227633266892 for osd.2 2026-03-31T21:14:23.151 INFO:teuthology.orchestra.run.vm00.stdout:163208757463 2026-03-31T21:14:23.160 INFO:teuthology.orchestra.run.vm00.stdout:206158430413 2026-03-31T21:14:23.163 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757464 got 163208757463 for osd.0 2026-03-31T21:14:23.175 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430414 got 206158430413 for osd.3 2026-03-31T21:14:23.197 INFO:teuthology.orchestra.run.vm00.stdout:90194313468 2026-03-31T21:14:23.209 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313470 got 90194313468 for osd.1 2026-03-31T21:14:24.124 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:24.163 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:24.175 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:24.210 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:24.330 INFO:teuthology.orchestra.run.vm00.stdout:227633266893 2026-03-31T21:14:24.352 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266893 got 227633266893 for osd.2 2026-03-31T21:14:24.352 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:24.394 INFO:teuthology.orchestra.run.vm00.stdout:163208757464 2026-03-31T21:14:24.405 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757464 got 163208757464 for osd.0 2026-03-31T21:14:24.405 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:24.429 INFO:teuthology.orchestra.run.vm00.stdout:206158430414 2026-03-31T21:14:24.440 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430414 got 206158430414 for osd.3 2026-03-31T21:14:24.440 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:24.445 INFO:teuthology.orchestra.run.vm00.stdout:90194313470 2026-03-31T21:14:24.457 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313470 got 90194313470 for osd.1 2026-03-31T21:14:24.457 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:24.457 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:24.691 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:24.691 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:24.702 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2026-03-31T21:14:23.895755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7589,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19267,"num_bytes_recovered":158108848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":327568,"kb_used_data":132056,"kb_used_omap":58,"kb_used_meta":195397,"kb_avail":282787952,"statfs":{"total":289910292480,"available":289574862848,"internally_reserved":0,"allocated":135225344,"data_stored":134351490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200087025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-130,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001574"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":52099,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:22.812866+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:22.812866+0000","last_peered":"2026-03-31T21:14:22.812866+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:22.812866+0000","last_undegraded":"2026-03-31T21:14:22.812866+0000","last_fullsized":"2026-03-31T21:14:22.812866+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7589,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19265,"num_bytes_recovered":157780656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7589,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19265,"num_bytes_recovered":157780656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187916288,"data_stored":187916288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430414,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266893,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313470,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757464,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53400,"kb_used_data":23880,"kb_used_omap":17,"kb_used_meta":29486,"kb_avail":94318440,"statfs":{"total":96636764160,"available":96582082560,"internally_reserved":0,"allocated":24453120,"data_stored":24159446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30193996},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23670784,"data_stored":23670784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:24.703 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 52099, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:22.812866+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:22.812866+0000', 'last_peered': '2026-03-31T21:14:22.812866+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:22.812866+0000', 'last_undegraded': '2026-03-31T21:14:22.812866+0000', 'last_fullsized': '2026-03-31T21:14:22.812866+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7589, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19265, 'num_bytes_recovered': 157780656, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:24.703 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:29.704 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:29.704 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:29.704 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:29.704 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:29.813 INFO:teuthology.orchestra.run.vm00.stdout:90194313472 2026-03-31T21:14:29.813 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:29.814 INFO:teuthology.orchestra.run.vm00.stdout:163208757467 2026-03-31T21:14:29.814 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:29.819 INFO:teuthology.orchestra.run.vm00.stdout:206158430417 2026-03-31T21:14:29.819 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:29.824 INFO:teuthology.orchestra.run.vm00.stdout:227633266895 2026-03-31T21:14:29.824 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:30.030 INFO:teuthology.orchestra.run.vm00.stdout:163208757465 2026-03-31T21:14:30.036 INFO:teuthology.orchestra.run.vm00.stdout:90194313471 2026-03-31T21:14:30.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757467 got 163208757465 for osd.0 2026-03-31T21:14:30.050 INFO:teuthology.orchestra.run.vm00.stdout:206158430415 2026-03-31T21:14:30.052 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313472 got 90194313471 for osd.1 2026-03-31T21:14:30.061 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430417 got 206158430415 for osd.3 2026-03-31T21:14:30.089 INFO:teuthology.orchestra.run.vm00.stdout:227633266894 2026-03-31T21:14:30.102 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266895 got 227633266894 for osd.2 2026-03-31T21:14:31.044 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:31.053 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:31.062 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:31.102 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:31.291 INFO:teuthology.orchestra.run.vm00.stdout:206158430417 2026-03-31T21:14:31.293 INFO:teuthology.orchestra.run.vm00.stdout:90194313472 2026-03-31T21:14:31.303 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430417 got 206158430417 for osd.3 2026-03-31T21:14:31.303 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:31.305 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313472 got 90194313472 for osd.1 2026-03-31T21:14:31.305 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:31.317 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:14:31.314+0000 7f13327fc640 0 --2- 192.168.123.100:0/114634737 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f1334057e90 0x7f133405b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:14:31.343 INFO:teuthology.orchestra.run.vm00.stdout:227633266895 2026-03-31T21:14:31.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266895 got 227633266895 for osd.2 2026-03-31T21:14:31.356 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:31.445 INFO:teuthology.orchestra.run.vm00.stdout:163208757467 2026-03-31T21:14:31.458 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757467 got 163208757467 for osd.0 2026-03-31T21:14:31.458 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:31.458 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:31.653 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:31.653 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:31.664 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2026-03-31T21:14:29.896445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7521,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19335,"num_bytes_recovered":158665904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":328036,"kb_used_data":132332,"kb_used_omap":58,"kb_used_meta":195589,"kb_avail":282787484,"statfs":{"total":289910292480,"available":289574383616,"internally_reserved":0,"allocated":135507968,"data_stored":134634114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":200283633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-148,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001455"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":52235,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:29.791974+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:29.791974+0000","last_peered":"2026-03-31T21:14:29.791974+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:29.791974+0000","last_undegraded":"2026-03-31T21:14:29.791974+0000","last_fullsized":"2026-03-31T21:14:29.791974+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7521,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19333,"num_bytes_recovered":158337712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7521,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19333,"num_bytes_recovered":158337712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188194816,"data_stored":188194816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430417,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266895,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757467,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53868,"kb_used_data":24156,"kb_used_omap":17,"kb_used_meta":29678,"kb_avail":94317972,"statfs":{"total":96636764160,"available":96581603328,"internally_reserved":0,"allocated":24735744,"data_stored":24442070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":30390604},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":23949312,"data_stored":23949312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:31.664 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 52235, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:29.791974+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:29.791974+0000', 'last_peered': '2026-03-31T21:14:29.791974+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:29.791974+0000', 'last_undegraded': '2026-03-31T21:14:29.791974+0000', 'last_fullsized': '2026-03-31T21:14:29.791974+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7521, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19333, 'num_bytes_recovered': 158337712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:31.664 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:36.665 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:36.665 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:36.665 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:36.665 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:36.778 INFO:teuthology.orchestra.run.vm00.stdout:163208757469 2026-03-31T21:14:36.778 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:36.783 INFO:teuthology.orchestra.run.vm00.stdout:227633266898 2026-03-31T21:14:36.783 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:36.789 INFO:teuthology.orchestra.run.vm00.stdout:206158430419 2026-03-31T21:14:36.790 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:36.974 INFO:teuthology.orchestra.run.vm00.stdout:90194313474 2026-03-31T21:14:36.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:37.033 INFO:teuthology.orchestra.run.vm00.stdout:206158430418 2026-03-31T21:14:37.037 INFO:teuthology.orchestra.run.vm00.stdout:227633266896 2026-03-31T21:14:37.049 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430419 got 206158430418 for osd.3 2026-03-31T21:14:37.052 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266898 got 227633266896 for osd.2 2026-03-31T21:14:37.056 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:14:37.053+0000 7f716ffff640 0 --2- 192.168.123.100:0/1622765272 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f717007ed90 0x7f717012c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:14:37.177 INFO:teuthology.orchestra.run.vm00.stdout:163208757468 2026-03-31T21:14:37.180 INFO:teuthology.orchestra.run.vm00.stdout:90194313473 2026-03-31T21:14:37.189 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757469 got 163208757468 for osd.0 2026-03-31T21:14:37.195 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313474 got 90194313473 for osd.1 2026-03-31T21:14:38.049 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:38.053 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:38.190 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:38.195 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:38.254 INFO:teuthology.orchestra.run.vm00.stdout:206158430418 2026-03-31T21:14:38.271 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430419 got 206158430418 for osd.3 2026-03-31T21:14:38.304 INFO:teuthology.orchestra.run.vm00.stdout:227633266898 2026-03-31T21:14:38.318 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266898 got 227633266898 for osd.2 2026-03-31T21:14:38.319 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:38.402 INFO:teuthology.orchestra.run.vm00.stdout:163208757469 2026-03-31T21:14:38.414 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757469 got 163208757469 for osd.0 2026-03-31T21:14:38.414 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:38.448 INFO:teuthology.orchestra.run.vm00.stdout:90194313475 2026-03-31T21:14:38.460 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313474 got 90194313475 for osd.1 2026-03-31T21:14:38.460 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:39.272 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:39.463 INFO:teuthology.orchestra.run.vm00.stdout:206158430419 2026-03-31T21:14:39.473 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430419 got 206158430419 for osd.3 2026-03-31T21:14:39.473 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:39.473 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:39.658 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:39.659 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:39.668 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2026-03-31T21:14:37.897553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7454,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19402,"num_bytes_recovered":159214768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":346168,"kb_used_data":132608,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282769352,"statfs":{"total":289910292480,"available":289555816448,"internally_reserved":0,"allocated":135790592,"data_stored":134916738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-120,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":52369,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:36.672689+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:36.672689+0000","last_peered":"2026-03-31T21:14:36.672689+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:36.672689+0000","last_undegraded":"2026-03-31T21:14:36.672689+0000","last_fullsized":"2026-03-31T21:14:36.672689+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7454,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19400,"num_bytes_recovered":158886576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7454,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19400,"num_bytes_recovered":158886576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188469248,"data_stored":188469248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266898,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313475,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757469,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72000,"kb_used_data":24432,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94299840,"statfs":{"total":96636764160,"available":96563036160,"internally_reserved":0,"allocated":25018368,"data_stored":24724694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24223744,"data_stored":24223744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:39.669 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 52369, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:36.672689+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:36.672689+0000', 'last_peered': '2026-03-31T21:14:36.672689+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:36.672689+0000', 'last_undegraded': '2026-03-31T21:14:36.672689+0000', 'last_fullsized': '2026-03-31T21:14:36.672689+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7454, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19400, 'num_bytes_recovered': 158886576, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:39.669 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:44.670 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:44.670 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:44.670 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:44.671 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:44.781 INFO:teuthology.orchestra.run.vm00.stdout:227633266900 2026-03-31T21:14:44.782 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:44.783 INFO:teuthology.orchestra.run.vm00.stdout:206158430422 2026-03-31T21:14:44.783 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:44.790 INFO:teuthology.orchestra.run.vm00.stdout:90194313477 2026-03-31T21:14:44.791 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:44.792 INFO:teuthology.orchestra.run.vm00.stdout:163208757472 2026-03-31T21:14:44.792 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:44.997 INFO:teuthology.orchestra.run.vm00.stdout:206158430420 2026-03-31T21:14:44.999 INFO:teuthology.orchestra.run.vm00.stdout:90194313476 2026-03-31T21:14:45.005 INFO:teuthology.orchestra.run.vm00.stdout:227633266899 2026-03-31T21:14:45.012 INFO:teuthology.orchestra.run.vm00.stdout:163208757470 2026-03-31T21:14:45.012 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313477 got 90194313476 for osd.1 2026-03-31T21:14:45.014 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430422 got 206158430420 for osd.3 2026-03-31T21:14:45.022 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266900 got 227633266899 for osd.2 2026-03-31T21:14:45.023 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757472 got 163208757470 for osd.0 2026-03-31T21:14:46.013 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:46.014 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:46.024 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:46.024 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:46.224 INFO:teuthology.orchestra.run.vm00.stdout:90194313477 2026-03-31T21:14:46.234 INFO:teuthology.orchestra.run.vm00.stdout:206158430423 2026-03-31T21:14:46.241 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313477 got 90194313477 for osd.1 2026-03-31T21:14:46.241 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:46.249 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430422 got 206158430423 for osd.3 2026-03-31T21:14:46.249 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:46.272 INFO:teuthology.orchestra.run.vm00.stdout:163208757472 2026-03-31T21:14:46.283 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757472 got 163208757472 for osd.0 2026-03-31T21:14:46.283 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:46.285 INFO:teuthology.orchestra.run.vm00.stdout:227633266900 2026-03-31T21:14:46.297 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266900 got 227633266900 for osd.2 2026-03-31T21:14:46.297 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:46.297 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:46.525 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:46.525 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:46.537 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2026-03-31T21:14:45.898509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7376,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19480,"num_bytes_recovered":159853744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":346480,"kb_used_data":132920,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282769040,"statfs":{"total":289910292480,"available":289555496960,"internally_reserved":0,"allocated":136110080,"data_stored":135236226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":52525,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:44.722975+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:44.722975+0000","last_peered":"2026-03-31T21:14:44.722975+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:44.722975+0000","last_undegraded":"2026-03-31T21:14:44.722975+0000","last_fullsized":"2026-03-31T21:14:44.722975+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7376,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19478,"num_bytes_recovered":159525552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7376,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19478,"num_bytes_recovered":159525552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188788736,"data_stored":188788736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266900,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757472,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72312,"kb_used_data":24744,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94299528,"statfs":{"total":96636764160,"available":96562716672,"internally_reserved":0,"allocated":25337856,"data_stored":25044182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24543232,"data_stored":24543232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:46.538 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 52525, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:44.722975+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:44.722975+0000', 'last_peered': '2026-03-31T21:14:44.722975+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:44.722975+0000', 'last_undegraded': '2026-03-31T21:14:44.722975+0000', 'last_fullsized': '2026-03-31T21:14:44.722975+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7376, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19478, 'num_bytes_recovered': 159525552, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:46.538 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:51.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:51.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:51.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:51.539 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:51.653 INFO:teuthology.orchestra.run.vm00.stdout:90194313479 2026-03-31T21:14:51.653 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:51.654 INFO:teuthology.orchestra.run.vm00.stdout:206158430425 2026-03-31T21:14:51.654 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:51.655 INFO:teuthology.orchestra.run.vm00.stdout:227633266902 2026-03-31T21:14:51.655 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:51.663 INFO:teuthology.orchestra.run.vm00.stdout:163208757474 2026-03-31T21:14:51.663 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:51.861 INFO:teuthology.orchestra.run.vm00.stdout:227633266901 2026-03-31T21:14:51.867 INFO:teuthology.orchestra.run.vm00.stdout:90194313478 2026-03-31T21:14:51.874 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266902 got 227633266901 for osd.2 2026-03-31T21:14:51.879 INFO:teuthology.orchestra.run.vm00.stdout:206158430424 2026-03-31T21:14:51.889 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313479 got 90194313478 for osd.1 2026-03-31T21:14:51.894 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430425 got 206158430424 for osd.3 2026-03-31T21:14:51.937 INFO:teuthology.orchestra.run.vm00.stdout:163208757473 2026-03-31T21:14:51.949 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757474 got 163208757473 for osd.0 2026-03-31T21:14:52.875 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:52.890 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:52.894 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:52.950 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:53.099 INFO:teuthology.orchestra.run.vm00.stdout:227633266903 2026-03-31T21:14:53.116 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266902 got 227633266903 for osd.2 2026-03-31T21:14:53.116 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:53.141 INFO:teuthology.orchestra.run.vm00.stdout:206158430425 2026-03-31T21:14:53.141 INFO:teuthology.orchestra.run.vm00.stdout:90194313479 2026-03-31T21:14:53.153 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430425 got 206158430425 for osd.3 2026-03-31T21:14:53.154 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:53.160 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313479 got 90194313479 for osd.1 2026-03-31T21:14:53.160 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:53.184 INFO:teuthology.orchestra.run.vm00.stdout:163208757474 2026-03-31T21:14:53.195 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757474 got 163208757474 for osd.0 2026-03-31T21:14:53.195 DEBUG:teuthology.parallel:result is None 2026-03-31T21:14:53.195 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:14:53.427 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:14:53.427 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:14:53.437 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2026-03-31T21:14:51.899217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7309,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19547,"num_bytes_recovered":160402608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":346748,"kb_used_data":133188,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282768772,"statfs":{"total":289910292480,"available":289555222528,"internally_reserved":0,"allocated":136384512,"data_stored":135510658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-119,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":52659,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:51.634966+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:51.634966+0000","last_peered":"2026-03-31T21:14:51.634966+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:51.634966+0000","last_undegraded":"2026-03-31T21:14:51.634966+0000","last_fullsized":"2026-03-31T21:14:51.634966+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7309,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19545,"num_bytes_recovered":160074416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7309,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19545,"num_bytes_recovered":160074416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":189063168,"data_stored":189063168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430425,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266903,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757474,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72580,"kb_used_data":25012,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94299260,"statfs":{"total":96636764160,"available":96562442240,"internally_reserved":0,"allocated":25612288,"data_stored":25318614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24817664,"data_stored":24817664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:14:53.437 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 52659, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:51.634966+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:51.634966+0000', 'last_peered': '2026-03-31T21:14:51.634966+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:51.634966+0000', 'last_undegraded': '2026-03-31T21:14:51.634966+0000', 'last_fullsized': '2026-03-31T21:14:51.634966+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7309, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19545, 'num_bytes_recovered': 160074416, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:14:53.437 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:14:58.438 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:14:58.439 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:14:58.439 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:14:58.439 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:14:58.562 INFO:teuthology.orchestra.run.vm00.stdout:90194313482 2026-03-31T21:14:58.562 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:58.565 INFO:teuthology.orchestra.run.vm00.stdout:227633266905 2026-03-31T21:14:58.565 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:58.584 INFO:teuthology.orchestra.run.vm00.stdout:163208757476 2026-03-31T21:14:58.584 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:14:58.595 INFO:teuthology.orchestra.run.vm00.stdout:206158430427 2026-03-31T21:14:58.595 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:58.817 INFO:teuthology.orchestra.run.vm00.stdout:90194313481 2026-03-31T21:14:58.834 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313482 got 90194313481 for osd.1 2026-03-31T21:14:58.847 INFO:teuthology.orchestra.run.vm00.stdout:227633266904 2026-03-31T21:14:58.864 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266905 got 227633266904 for osd.2 2026-03-31T21:14:58.882 INFO:teuthology.orchestra.run.vm00.stdout:206158430426 2026-03-31T21:14:58.897 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430427 got 206158430426 for osd.3 2026-03-31T21:14:58.902 INFO:teuthology.orchestra.run.vm00.stdout:163208757475 2026-03-31T21:14:58.914 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757476 got 163208757475 for osd.0 2026-03-31T21:14:59.836 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:14:59.864 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:14:59.898 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:14:59.916 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:00.081 INFO:teuthology.orchestra.run.vm00.stdout:90194313482 2026-03-31T21:15:00.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313482 got 90194313482 for osd.1 2026-03-31T21:15:00.104 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:00.134 INFO:teuthology.orchestra.run.vm00.stdout:227633266905 2026-03-31T21:15:00.154 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266905 got 227633266905 for osd.2 2026-03-31T21:15:00.154 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:00.162 INFO:teuthology.orchestra.run.vm00.stdout:206158430428 2026-03-31T21:15:00.174 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430427 got 206158430428 for osd.3 2026-03-31T21:15:00.174 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:00.182 INFO:teuthology.orchestra.run.vm00.stdout:163208757477 2026-03-31T21:15:00.194 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757476 got 163208757477 for osd.0 2026-03-31T21:15:00.194 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:00.194 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:00.453 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:00.453 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:00.465 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2026-03-31T21:14:59.900288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7235,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19621,"num_bytes_recovered":161008816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":347044,"kb_used_data":133484,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282768476,"statfs":{"total":289910292480,"available":289554919424,"internally_reserved":0,"allocated":136687616,"data_stored":135813762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-141,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001520"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":52807,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:14:59.711814+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:14:59.711814+0000","last_peered":"2026-03-31T21:14:59.711814+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:14:59.711814+0000","last_undegraded":"2026-03-31T21:14:59.711814+0000","last_fullsized":"2026-03-31T21:14:59.711814+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7235,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19619,"num_bytes_recovered":160680624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7235,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19619,"num_bytes_recovered":160680624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":189366272,"data_stored":189366272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430428,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266905,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313482,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757477,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72876,"kb_used_data":25308,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94298964,"statfs":{"total":96636764160,"available":96562139136,"internally_reserved":0,"allocated":25915392,"data_stored":25621718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25120768,"data_stored":25120768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:00.465 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 52807, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:14:59.711814+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:14:59.711814+0000', 'last_peered': '2026-03-31T21:14:59.711814+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:14:59.711814+0000', 'last_undegraded': '2026-03-31T21:14:59.711814+0000', 'last_fullsized': '2026-03-31T21:14:59.711814+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7235, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19619, 'num_bytes_recovered': 160680624, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:00.466 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:05.466 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:05.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:05.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:05.467 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:05.575 INFO:teuthology.orchestra.run.vm00.stdout:227633266907 2026-03-31T21:15:05.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:05.578 INFO:teuthology.orchestra.run.vm00.stdout:90194313484 2026-03-31T21:15:05.578 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:05.584 INFO:teuthology.orchestra.run.vm00.stdout:163208757479 2026-03-31T21:15:05.584 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:05.589 INFO:teuthology.orchestra.run.vm00.stdout:206158430430 2026-03-31T21:15:05.589 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:05.805 INFO:teuthology.orchestra.run.vm00.stdout:227633266906 2026-03-31T21:15:05.809 INFO:teuthology.orchestra.run.vm00.stdout:163208757477 2026-03-31T21:15:05.809 INFO:teuthology.orchestra.run.vm00.stdout:90194313483 2026-03-31T21:15:05.822 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266907 got 227633266906 for osd.2 2026-03-31T21:15:05.823 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757479 got 163208757477 for osd.0 2026-03-31T21:15:05.823 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313484 got 90194313483 for osd.1 2026-03-31T21:15:05.869 INFO:teuthology.orchestra.run.vm00.stdout:206158430428 2026-03-31T21:15:05.882 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430430 got 206158430428 for osd.3 2026-03-31T21:15:06.822 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:06.823 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:06.824 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:06.882 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:07.053 INFO:teuthology.orchestra.run.vm00.stdout:163208757479 2026-03-31T21:15:07.058 INFO:teuthology.orchestra.run.vm00.stdout:90194313484 2026-03-31T21:15:07.072 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757479 got 163208757479 for osd.0 2026-03-31T21:15:07.072 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:07.075 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313484 got 90194313484 for osd.1 2026-03-31T21:15:07.075 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:07.083 INFO:teuthology.orchestra.run.vm00.stdout:227633266907 2026-03-31T21:15:07.095 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266907 got 227633266907 for osd.2 2026-03-31T21:15:07.095 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:07.133 INFO:teuthology.orchestra.run.vm00.stdout:206158430430 2026-03-31T21:15:07.147 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430430 got 206158430430 for osd.3 2026-03-31T21:15:07.147 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:07.147 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:07.382 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:07.382 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:07.394 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2026-03-31T21:15:05.900998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7182,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19674,"num_bytes_recovered":161442992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":347260,"kb_used_data":133700,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282768260,"statfs":{"total":289910292480,"available":289554698240,"internally_reserved":0,"allocated":136908800,"data_stored":136034946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":52913,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:05.498214+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:05.498214+0000","last_peered":"2026-03-31T21:15:05.498214+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:05.498214+0000","last_undegraded":"2026-03-31T21:15:05.498214+0000","last_fullsized":"2026-03-31T21:15:05.498214+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7182,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19672,"num_bytes_recovered":161114800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7182,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19672,"num_bytes_recovered":161114800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":189583360,"data_stored":189583360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430430,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266907,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757479,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73092,"kb_used_data":25524,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94298748,"statfs":{"total":96636764160,"available":96561917952,"internally_reserved":0,"allocated":26136576,"data_stored":25842902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25337856,"data_stored":25337856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:07.394 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 52913, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:05.498214+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:05.498214+0000', 'last_peered': '2026-03-31T21:15:05.498214+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:05.498214+0000', 'last_undegraded': '2026-03-31T21:15:05.498214+0000', 'last_fullsized': '2026-03-31T21:15:05.498214+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7182, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19672, 'num_bytes_recovered': 161114800, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:07.394 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:12.395 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:12.396 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:12.396 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:12.396 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:12.505 INFO:teuthology.orchestra.run.vm00.stdout:90194313486 2026-03-31T21:15:12.506 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:12.511 INFO:teuthology.orchestra.run.vm00.stdout:227633266910 2026-03-31T21:15:12.511 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:12.513 INFO:teuthology.orchestra.run.vm00.stdout:206158430432 2026-03-31T21:15:12.513 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:12.678 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:15:12.676+0000 7fac3e7fc640 0 --2- 192.168.123.100:0/654118027 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7fac4007f6f0 0x7fac2c026950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:15:12.714 INFO:teuthology.orchestra.run.vm00.stdout:163208757481 2026-03-31T21:15:12.714 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:12.726 INFO:teuthology.orchestra.run.vm00.stdout:227633266909 2026-03-31T21:15:12.729 INFO:teuthology.orchestra.run.vm00.stdout:90194313485 2026-03-31T21:15:12.733 INFO:teuthology.orchestra.run.vm00.stdout:206158430431 2026-03-31T21:15:12.742 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266910 got 227633266909 for osd.2 2026-03-31T21:15:12.743 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313486 got 90194313485 for osd.1 2026-03-31T21:15:12.747 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430432 got 206158430431 for osd.3 2026-03-31T21:15:12.919 INFO:teuthology.orchestra.run.vm00.stdout:163208757480 2026-03-31T21:15:12.931 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757481 got 163208757480 for osd.0 2026-03-31T21:15:13.743 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:13.743 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:13.748 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:13.932 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:13.952 INFO:teuthology.orchestra.run.vm00.stdout:90194313487 2026-03-31T21:15:13.957 INFO:teuthology.orchestra.run.vm00.stdout:227633266910 2026-03-31T21:15:13.971 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266910 got 227633266910 for osd.2 2026-03-31T21:15:13.972 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:13.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313486 got 90194313487 for osd.1 2026-03-31T21:15:13.973 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:14.001 INFO:teuthology.orchestra.run.vm00.stdout:206158430432 2026-03-31T21:15:14.013 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430432 got 206158430432 for osd.3 2026-03-31T21:15:14.013 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:14.150 INFO:teuthology.orchestra.run.vm00.stdout:163208757481 2026-03-31T21:15:14.162 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757481 got 163208757481 for osd.0 2026-03-31T21:15:14.162 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:14.162 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:14.437 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:15:14.435+0000 7f615cc36640 0 --2- 192.168.123.100:0/2853413539 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f61581734f0 0x7f61581938d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:15:14.565 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:14.565 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:14.577 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2026-03-31T21:15:13.901917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7119,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19737,"num_bytes_recovered":161959088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":347520,"kb_used_data":133960,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282768000,"statfs":{"total":289910292480,"available":289554432000,"internally_reserved":0,"allocated":137175040,"data_stored":136301186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-116,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001397"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":53039,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:12.407108+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:12.407108+0000","last_peered":"2026-03-31T21:15:12.407108+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:12.407108+0000","last_undegraded":"2026-03-31T21:15:12.407108+0000","last_fullsized":"2026-03-31T21:15:12.407108+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19735,"num_bytes_recovered":161630896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19735,"num_bytes_recovered":161630896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":189849600,"data_stored":189849600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430432,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266910,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313487,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73352,"kb_used_data":25784,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94298488,"statfs":{"total":96636764160,"available":96561651712,"internally_reserved":0,"allocated":26402816,"data_stored":26109142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25604096,"data_stored":25604096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:14.578 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 53039, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:12.407108+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:12.407108+0000', 'last_peered': '2026-03-31T21:15:12.407108+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:12.407108+0000', 'last_undegraded': '2026-03-31T21:15:12.407108+0000', 'last_fullsized': '2026-03-31T21:15:12.407108+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7119, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19735, 'num_bytes_recovered': 161630896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:14.578 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:19.578 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:19.579 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:19.579 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:19.579 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:19.684 INFO:teuthology.orchestra.run.vm00.stdout:206158430434 2026-03-31T21:15:19.684 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:19.689 INFO:teuthology.orchestra.run.vm00.stdout:163208757483 2026-03-31T21:15:19.689 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:19.695 INFO:teuthology.orchestra.run.vm00.stdout:227633266912 2026-03-31T21:15:19.695 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:19.856 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:15:19.854+0000 7f7fe3577640 0 --2- 192.168.123.100:0/1874443950 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f7fe4172e20 0x7f7fe4193200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:15:19.887 INFO:teuthology.orchestra.run.vm00.stdout:90194313489 2026-03-31T21:15:19.887 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:19.895 INFO:teuthology.orchestra.run.vm00.stdout:206158430433 2026-03-31T21:15:19.912 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430434 got 206158430433 for osd.3 2026-03-31T21:15:19.951 INFO:teuthology.orchestra.run.vm00.stdout:227633266912 2026-03-31T21:15:19.953 INFO:teuthology.orchestra.run.vm00.stdout:163208757484 2026-03-31T21:15:19.963 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266912 got 227633266912 for osd.2 2026-03-31T21:15:19.963 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:19.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757483 got 163208757484 for osd.0 2026-03-31T21:15:19.964 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:20.088 INFO:teuthology.orchestra.run.vm00.stdout:90194313489 2026-03-31T21:15:20.099 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313489 got 90194313489 for osd.1 2026-03-31T21:15:20.099 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:20.912 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:21.110 INFO:teuthology.orchestra.run.vm00.stdout:206158430435 2026-03-31T21:15:21.121 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430434 got 206158430435 for osd.3 2026-03-31T21:15:21.121 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:21.121 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:21.307 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:21.307 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:21.319 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2026-03-31T21:15:19.902597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7048,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19808,"num_bytes_recovered":162540720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":347784,"kb_used_data":134224,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282767736,"statfs":{"total":289910292480,"available":289554161664,"internally_reserved":0,"allocated":137445376,"data_stored":136571522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-134,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001327"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":53181,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:19.675081+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:19.675081+0000","last_peered":"2026-03-31T21:15:19.675081+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:19.675081+0000","last_undegraded":"2026-03-31T21:15:19.675081+0000","last_fullsized":"2026-03-31T21:15:19.675081+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7048,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19806,"num_bytes_recovered":162212528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7048,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19806,"num_bytes_recovered":162212528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190132224,"data_stored":190132224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430435,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266912,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757484,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73616,"kb_used_data":26048,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94298224,"statfs":{"total":96636764160,"available":96561381376,"internally_reserved":0,"allocated":26673152,"data_stored":26379478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":25886720,"data_stored":25886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:21.319 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 53181, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:19.675081+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:19.675081+0000', 'last_peered': '2026-03-31T21:15:19.675081+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:19.675081+0000', 'last_undegraded': '2026-03-31T21:15:19.675081+0000', 'last_fullsized': '2026-03-31T21:15:19.675081+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7048, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19806, 'num_bytes_recovered': 162212528, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:21.319 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:26.320 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:26.321 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:26.321 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:26.321 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:26.423 INFO:teuthology.orchestra.run.vm00.stdout:206158430437 2026-03-31T21:15:26.423 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:26.424 INFO:teuthology.orchestra.run.vm00.stdout:227633266914 2026-03-31T21:15:26.424 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:26.425 INFO:teuthology.orchestra.run.vm00.stdout:90194313491 2026-03-31T21:15:26.425 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:26.426 INFO:teuthology.orchestra.run.vm00.stdout:163208757486 2026-03-31T21:15:26.426 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:26.629 INFO:teuthology.orchestra.run.vm00.stdout:206158430436 2026-03-31T21:15:26.630 INFO:teuthology.orchestra.run.vm00.stdout:227633266913 2026-03-31T21:15:26.646 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430437 got 206158430436 for osd.3 2026-03-31T21:15:26.646 INFO:teuthology.orchestra.run.vm00.stdout:90194313490 2026-03-31T21:15:26.647 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266914 got 227633266913 for osd.2 2026-03-31T21:15:26.649 INFO:teuthology.orchestra.run.vm00.stdout:163208757485 2026-03-31T21:15:26.657 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313491 got 90194313490 for osd.1 2026-03-31T21:15:26.661 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757486 got 163208757485 for osd.0 2026-03-31T21:15:27.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:27.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:27.657 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:27.662 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:27.855 INFO:teuthology.orchestra.run.vm00.stdout:227633266913 2026-03-31T21:15:27.871 INFO:teuthology.orchestra.run.vm00.stdout:206158430436 2026-03-31T21:15:27.873 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266914 got 227633266913 for osd.2 2026-03-31T21:15:27.886 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430437 got 206158430436 for osd.3 2026-03-31T21:15:27.922 INFO:teuthology.orchestra.run.vm00.stdout:90194313490 2026-03-31T21:15:27.936 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313491 got 90194313490 for osd.1 2026-03-31T21:15:27.940 INFO:teuthology.orchestra.run.vm00.stdout:163208757485 2026-03-31T21:15:27.952 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757486 got 163208757485 for osd.0 2026-03-31T21:15:28.873 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:28.887 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:28.936 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:28.953 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:29.076 INFO:teuthology.orchestra.run.vm00.stdout:227633266915 2026-03-31T21:15:29.094 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266914 got 227633266915 for osd.2 2026-03-31T21:15:29.094 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:29.137 INFO:teuthology.orchestra.run.vm00.stdout:206158430437 2026-03-31T21:15:29.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430437 got 206158430437 for osd.3 2026-03-31T21:15:29.148 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:29.173 INFO:teuthology.orchestra.run.vm00.stdout:163208757486 2026-03-31T21:15:29.184 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757486 got 163208757486 for osd.0 2026-03-31T21:15:29.184 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:29.185 INFO:teuthology.orchestra.run.vm00.stdout:90194313492 2026-03-31T21:15:29.197 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313491 got 90194313492 for osd.1 2026-03-31T21:15:29.197 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:29.197 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:29.425 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:29.425 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:29.436 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2026-03-31T21:15:27.903558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6983,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19873,"num_bytes_recovered":163073200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":348056,"kb_used_data":134496,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282767464,"statfs":{"total":289910292480,"available":289553883136,"internally_reserved":0,"allocated":137723904,"data_stored":136850050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-113,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001393"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":53311,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:26.319393+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:26.319393+0000","last_peered":"2026-03-31T21:15:26.319393+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:26.319393+0000","last_undegraded":"2026-03-31T21:15:26.319393+0000","last_fullsized":"2026-03-31T21:15:26.319393+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6983,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19871,"num_bytes_recovered":162745008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6983,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19871,"num_bytes_recovered":162745008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190398464,"data_stored":190398464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430437,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266915,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313492,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757486,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73888,"kb_used_data":26320,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94297952,"statfs":{"total":96636764160,"available":96561102848,"internally_reserved":0,"allocated":26951680,"data_stored":26658006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26152960,"data_stored":26152960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:29.436 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 53311, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:26.319393+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:26.319393+0000', 'last_peered': '2026-03-31T21:15:26.319393+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:26.319393+0000', 'last_undegraded': '2026-03-31T21:15:26.319393+0000', 'last_fullsized': '2026-03-31T21:15:26.319393+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6983, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19871, 'num_bytes_recovered': 162745008, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:29.436 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:34.437 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:34.437 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:34.437 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:34.438 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:34.549 INFO:teuthology.orchestra.run.vm00.stdout:227633266917 2026-03-31T21:15:34.549 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:34.553 INFO:teuthology.orchestra.run.vm00.stdout:90194313494 2026-03-31T21:15:34.553 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:34.558 INFO:teuthology.orchestra.run.vm00.stdout:206158430439 2026-03-31T21:15:34.558 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:34.749 INFO:teuthology.orchestra.run.vm00.stdout:163208757488 2026-03-31T21:15:34.749 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:34.754 INFO:teuthology.orchestra.run.vm00.stdout:227633266916 2026-03-31T21:15:34.766 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266917 got 227633266916 for osd.2 2026-03-31T21:15:34.800 INFO:teuthology.orchestra.run.vm00.stdout:90194313493 2026-03-31T21:15:34.804 INFO:teuthology.orchestra.run.vm00.stdout:206158430438 2026-03-31T21:15:34.813 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313494 got 90194313493 for osd.1 2026-03-31T21:15:34.817 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430439 got 206158430438 for osd.3 2026-03-31T21:15:34.947 INFO:teuthology.orchestra.run.vm00.stdout:163208757487 2026-03-31T21:15:34.958 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757488 got 163208757487 for osd.0 2026-03-31T21:15:35.767 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:35.814 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:35.818 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:35.959 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:35.962 INFO:teuthology.orchestra.run.vm00.stdout:227633266916 2026-03-31T21:15:35.975 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266917 got 227633266916 for osd.2 2026-03-31T21:15:36.016 INFO:teuthology.orchestra.run.vm00.stdout:90194313494 2026-03-31T21:15:36.033 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313494 got 90194313494 for osd.1 2026-03-31T21:15:36.034 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:36.052 INFO:teuthology.orchestra.run.vm00.stdout:206158430440 2026-03-31T21:15:36.063 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430439 got 206158430440 for osd.3 2026-03-31T21:15:36.063 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:36.163 INFO:teuthology.orchestra.run.vm00.stdout:163208757489 2026-03-31T21:15:36.173 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757488 got 163208757489 for osd.0 2026-03-31T21:15:36.173 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:36.976 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:37.174 INFO:teuthology.orchestra.run.vm00.stdout:227633266917 2026-03-31T21:15:37.184 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266917 got 227633266917 for osd.2 2026-03-31T21:15:37.184 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:37.184 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:37.367 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:37.367 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:37.377 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2026-03-31T21:15:35.904528+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6901,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19955,"num_bytes_recovered":163744944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":348368,"kb_used_data":134808,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282767152,"statfs":{"total":289910292480,"available":289553563648,"internally_reserved":0,"allocated":138043392,"data_stored":137169538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":53475,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:34.708304+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:34.708304+0000","last_peered":"2026-03-31T21:15:34.708304+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:34.708304+0000","last_undegraded":"2026-03-31T21:15:34.708304+0000","last_fullsized":"2026-03-31T21:15:34.708304+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19953,"num_bytes_recovered":163416752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":54,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:11:37.566545+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:11:37.566545+0000","last_peered":"2026-03-31T21:11:37.566545+0000","last_clean":"2026-03-31T21:11:37.566545+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:11:37.566545+0000","last_undegraded":"2026-03-31T21:11:37.566545+0000","last_fullsized":"2026-03-31T21:11:37.566545+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19953,"num_bytes_recovered":163416752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190734336,"data_stored":190734336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266917,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757489,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74200,"kb_used_data":26632,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94297640,"statfs":{"total":96636764160,"available":96560783360,"internally_reserved":0,"allocated":27271168,"data_stored":26977494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26488832,"data_stored":26488832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:37.378 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 53475, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:34.708304+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:34.708304+0000', 'last_peered': '2026-03-31T21:15:34.708304+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:34.708304+0000', 'last_undegraded': '2026-03-31T21:15:34.708304+0000', 'last_fullsized': '2026-03-31T21:15:34.708304+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6901, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 19953, 'num_bytes_recovered': 163416752, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:37.378 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:42.378 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:42.379 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:42.379 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:42.379 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:42.488 INFO:teuthology.orchestra.run.vm00.stdout:163208757491 2026-03-31T21:15:42.488 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:42.490 INFO:teuthology.orchestra.run.vm00.stdout:90194313496 2026-03-31T21:15:42.490 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:42.490 INFO:teuthology.orchestra.run.vm00.stdout:227633266920 2026-03-31T21:15:42.490 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:42.492 INFO:teuthology.orchestra.run.vm00.stdout:206158430442 2026-03-31T21:15:42.492 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:42.704 INFO:teuthology.orchestra.run.vm00.stdout:90194313495 2026-03-31T21:15:42.707 INFO:teuthology.orchestra.run.vm00.stdout:227633266919 2026-03-31T21:15:42.710 INFO:teuthology.orchestra.run.vm00.stdout:206158430441 2026-03-31T21:15:42.716 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313496 got 90194313495 for osd.1 2026-03-31T21:15:42.722 INFO:teuthology.orchestra.run.vm00.stdout:163208757490 2026-03-31T21:15:42.724 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266920 got 227633266919 for osd.2 2026-03-31T21:15:42.726 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430442 got 206158430441 for osd.3 2026-03-31T21:15:42.733 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757491 got 163208757490 for osd.0 2026-03-31T21:15:43.717 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:43.725 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:43.727 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:43.734 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:43.933 INFO:teuthology.orchestra.run.vm00.stdout:90194313495 2026-03-31T21:15:43.949 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313496 got 90194313495 for osd.1 2026-03-31T21:15:43.975 INFO:teuthology.orchestra.run.vm00.stdout:163208757490 2026-03-31T21:15:43.976 INFO:teuthology.orchestra.run.vm00.stdout:227633266919 2026-03-31T21:15:43.984 INFO:teuthology.orchestra.run.vm00.stdout:206158430441 2026-03-31T21:15:43.989 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757491 got 163208757490 for osd.0 2026-03-31T21:15:43.994 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266920 got 227633266919 for osd.2 2026-03-31T21:15:43.998 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430442 got 206158430441 for osd.3 2026-03-31T21:15:44.950 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:44.989 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:44.994 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:44.998 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:45.164 INFO:teuthology.orchestra.run.vm00.stdout:90194313497 2026-03-31T21:15:45.179 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313496 got 90194313497 for osd.1 2026-03-31T21:15:45.179 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:45.222 INFO:teuthology.orchestra.run.vm00.stdout:163208757491 2026-03-31T21:15:45.233 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757491 got 163208757491 for osd.0 2026-03-31T21:15:45.233 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:45.247 INFO:teuthology.orchestra.run.vm00.stdout:227633266920 2026-03-31T21:15:45.258 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266920 got 227633266920 for osd.2 2026-03-31T21:15:45.258 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:45.319 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:15:45.317+0000 7fd2077fe640 0 --2- 192.168.123.100:0/1832053081 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7fd2081704c0 0x7fd2081908a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:15:45.441 INFO:teuthology.orchestra.run.vm00.stdout:206158430442 2026-03-31T21:15:45.453 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430442 got 206158430442 for osd.3 2026-03-31T21:15:45.453 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:45.453 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:45.643 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:45.643 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:45.655 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2026-03-31T21:15:43.905529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6825,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20031,"num_bytes_recovered":164367536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":348680,"kb_used_data":135120,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282766840,"statfs":{"total":289910292480,"available":289553244160,"internally_reserved":0,"allocated":138362880,"data_stored":137489026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-125,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":53627,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:42.446384+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:42.446384+0000","last_peered":"2026-03-31T21:15:42.446384+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:42.446384+0000","last_undegraded":"2026-03-31T21:15:42.446384+0000","last_fullsized":"2026-03-31T21:15:42.446384+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6825,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20029,"num_bytes_recovered":164039344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6825,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20029,"num_bytes_recovered":164039344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":191045632,"data_stored":191045632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266920,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313497,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757491,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74512,"kb_used_data":26944,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94297328,"statfs":{"total":96636764160,"available":96560463872,"internally_reserved":0,"allocated":27590656,"data_stored":27296982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26800128,"data_stored":26800128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:45.655 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 53627, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:42.446384+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:42.446384+0000', 'last_peered': '2026-03-31T21:15:42.446384+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:42.446384+0000', 'last_undegraded': '2026-03-31T21:15:42.446384+0000', 'last_fullsized': '2026-03-31T21:15:42.446384+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6825, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20029, 'num_bytes_recovered': 164039344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:45.655 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:50.656 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:50.656 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:50.656 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:50.656 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:50.764 INFO:teuthology.orchestra.run.vm00.stdout:227633266922 2026-03-31T21:15:50.764 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:50.764 INFO:teuthology.orchestra.run.vm00.stdout:163208757494 2026-03-31T21:15:50.765 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:50.771 INFO:teuthology.orchestra.run.vm00.stdout:90194313499 2026-03-31T21:15:50.771 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:50.772 INFO:teuthology.orchestra.run.vm00.stdout:206158430445 2026-03-31T21:15:50.772 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:50.989 INFO:teuthology.orchestra.run.vm00.stdout:90194313498 2026-03-31T21:15:51.001 INFO:teuthology.orchestra.run.vm00.stdout:163208757493 2026-03-31T21:15:51.001 INFO:teuthology.orchestra.run.vm00.stdout:227633266921 2026-03-31T21:15:51.008 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313499 got 90194313498 for osd.1 2026-03-31T21:15:51.017 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266922 got 227633266921 for osd.2 2026-03-31T21:15:51.018 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757494 got 163208757493 for osd.0 2026-03-31T21:15:51.040 INFO:teuthology.orchestra.run.vm00.stdout:206158430444 2026-03-31T21:15:51.051 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430445 got 206158430444 for osd.3 2026-03-31T21:15:52.009 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:52.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:52.019 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:52.052 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:52.232 INFO:teuthology.orchestra.run.vm00.stdout:90194313499 2026-03-31T21:15:52.249 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313499 got 90194313499 for osd.1 2026-03-31T21:15:52.249 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:52.274 INFO:teuthology.orchestra.run.vm00.stdout:206158430445 2026-03-31T21:15:52.278 INFO:teuthology.orchestra.run.vm00.stdout:163208757494 2026-03-31T21:15:52.289 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430445 got 206158430445 for osd.3 2026-03-31T21:15:52.289 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:52.295 INFO:teuthology.orchestra.run.vm00.stdout:227633266923 2026-03-31T21:15:52.296 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757494 got 163208757494 for osd.0 2026-03-31T21:15:52.297 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:52.306 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266922 got 227633266923 for osd.2 2026-03-31T21:15:52.307 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:52.307 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:52.540 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:52.540 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:52.552 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2026-03-31T21:15:51.906500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6744,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20112,"num_bytes_recovered":165031088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":349000,"kb_used_data":135440,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282766520,"statfs":{"total":289910292480,"available":289552916480,"internally_reserved":0,"allocated":138690560,"data_stored":137816706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":884736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001507"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":53789,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:50.697611+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:50.697611+0000","last_peered":"2026-03-31T21:15:50.697611+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:50.697611+0000","last_undegraded":"2026-03-31T21:15:50.697611+0000","last_fullsized":"2026-03-31T21:15:50.697611+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6744,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20110,"num_bytes_recovered":164702896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6744,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20110,"num_bytes_recovered":164702896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":191377408,"data_stored":191377408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430445,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757494,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74832,"kb_used_data":27264,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94297008,"statfs":{"total":96636764160,"available":96560136192,"internally_reserved":0,"allocated":27918336,"data_stored":27624662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27131904,"data_stored":27131904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:52.552 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 53789, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:50.697611+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:50.697611+0000', 'last_peered': '2026-03-31T21:15:50.697611+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:50.697611+0000', 'last_undegraded': '2026-03-31T21:15:50.697611+0000', 'last_fullsized': '2026-03-31T21:15:50.697611+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6744, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20110, 'num_bytes_recovered': 164702896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:52.552 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:15:57.553 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:15:57.553 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:15:57.553 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:15:57.553 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:15:57.662 INFO:teuthology.orchestra.run.vm00.stdout:206158430447 2026-03-31T21:15:57.662 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:57.671 INFO:teuthology.orchestra.run.vm00.stdout:90194313502 2026-03-31T21:15:57.671 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:57.678 INFO:teuthology.orchestra.run.vm00.stdout:163208757496 2026-03-31T21:15:57.678 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:57.679 INFO:teuthology.orchestra.run.vm00.stdout:227633266925 2026-03-31T21:15:57.679 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:57.937 INFO:teuthology.orchestra.run.vm00.stdout:227633266923 2026-03-31T21:15:57.951 INFO:teuthology.orchestra.run.vm00.stdout:90194313500 2026-03-31T21:15:57.951 INFO:teuthology.orchestra.run.vm00.stdout:206158430446 2026-03-31T21:15:57.955 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266925 got 227633266923 for osd.2 2026-03-31T21:15:57.962 INFO:teuthology.orchestra.run.vm00.stdout:163208757495 2026-03-31T21:15:57.965 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430447 got 206158430446 for osd.3 2026-03-31T21:15:57.965 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313502 got 90194313500 for osd.1 2026-03-31T21:15:57.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757496 got 163208757495 for osd.0 2026-03-31T21:15:58.955 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:15:58.965 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:15:58.965 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:15:58.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:15:59.177 INFO:teuthology.orchestra.run.vm00.stdout:227633266925 2026-03-31T21:15:59.194 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266925 got 227633266925 for osd.2 2026-03-31T21:15:59.194 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:59.226 INFO:teuthology.orchestra.run.vm00.stdout:163208757496 2026-03-31T21:15:59.230 INFO:teuthology.orchestra.run.vm00.stdout:206158430447 2026-03-31T21:15:59.243 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757496 got 163208757496 for osd.0 2026-03-31T21:15:59.243 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:59.248 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430447 got 206158430447 for osd.3 2026-03-31T21:15:59.248 INFO:teuthology.orchestra.run.vm00.stdout:90194313502 2026-03-31T21:15:59.248 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:59.259 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313502 got 90194313502 for osd.1 2026-03-31T21:15:59.259 DEBUG:teuthology.parallel:result is None 2026-03-31T21:15:59.259 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:15:59.490 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:15:59.491 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:15:59.502 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2026-03-31T21:15:57.907169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6677,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20179,"num_bytes_recovered":165579952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":349264,"kb_used_data":135704,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282766256,"statfs":{"total":289910292480,"available":289552646144,"internally_reserved":0,"allocated":138960896,"data_stored":138087042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001385"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":53923,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:15:57.577301+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:15:57.577301+0000","last_peered":"2026-03-31T21:15:57.577301+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:15:57.577301+0000","last_undegraded":"2026-03-31T21:15:57.577301+0000","last_fullsized":"2026-03-31T21:15:57.577301+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20177,"num_bytes_recovered":165251760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20177,"num_bytes_recovered":165251760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":191651840,"data_stored":191651840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266925,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313502,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757496,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75096,"kb_used_data":27528,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94296744,"statfs":{"total":96636764160,"available":96559865856,"internally_reserved":0,"allocated":28188672,"data_stored":27894998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27406336,"data_stored":27406336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:15:59.502 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 53923, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:15:57.577301+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:15:57.577301+0000', 'last_peered': '2026-03-31T21:15:57.577301+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:15:57.577301+0000', 'last_undegraded': '2026-03-31T21:15:57.577301+0000', 'last_fullsized': '2026-03-31T21:15:57.577301+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6677, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20177, 'num_bytes_recovered': 165251760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:15:59.502 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:04.503 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:04.503 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:04.504 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:04.504 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:04.612 INFO:teuthology.orchestra.run.vm00.stdout:206158430449 2026-03-31T21:16:04.613 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:04.613 INFO:teuthology.orchestra.run.vm00.stdout:227633266927 2026-03-31T21:16:04.613 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:04.614 INFO:teuthology.orchestra.run.vm00.stdout:90194313504 2026-03-31T21:16:04.614 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:04.621 INFO:teuthology.orchestra.run.vm00.stdout:163208757498 2026-03-31T21:16:04.621 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:04.825 INFO:teuthology.orchestra.run.vm00.stdout:227633266926 2026-03-31T21:16:04.831 INFO:teuthology.orchestra.run.vm00.stdout:90194313503 2026-03-31T21:16:04.840 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266927 got 227633266926 for osd.2 2026-03-31T21:16:04.842 INFO:teuthology.orchestra.run.vm00.stdout:206158430448 2026-03-31T21:16:04.854 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313504 got 90194313503 for osd.1 2026-03-31T21:16:04.860 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430449 got 206158430448 for osd.3 2026-03-31T21:16:04.887 INFO:teuthology.orchestra.run.vm00.stdout:163208757497 2026-03-31T21:16:04.898 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757498 got 163208757497 for osd.0 2026-03-31T21:16:05.841 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:05.855 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:05.861 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:05.899 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:06.058 INFO:teuthology.orchestra.run.vm00.stdout:227633266926 2026-03-31T21:16:06.070 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266927 got 227633266926 for osd.2 2026-03-31T21:16:06.098 INFO:teuthology.orchestra.run.vm00.stdout:90194313503 2026-03-31T21:16:06.103 INFO:teuthology.orchestra.run.vm00.stdout:206158430448 2026-03-31T21:16:06.110 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313504 got 90194313503 for osd.1 2026-03-31T21:16:06.115 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430449 got 206158430448 for osd.3 2026-03-31T21:16:06.159 INFO:teuthology.orchestra.run.vm00.stdout:163208757499 2026-03-31T21:16:06.170 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757498 got 163208757499 for osd.0 2026-03-31T21:16:06.170 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:07.070 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:07.111 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:07.116 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:07.286 INFO:teuthology.orchestra.run.vm00.stdout:227633266927 2026-03-31T21:16:07.298 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266927 got 227633266927 for osd.2 2026-03-31T21:16:07.298 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:07.335 INFO:teuthology.orchestra.run.vm00.stdout:90194313504 2026-03-31T21:16:07.338 INFO:teuthology.orchestra.run.vm00.stdout:206158430450 2026-03-31T21:16:07.349 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313504 got 90194313504 for osd.1 2026-03-31T21:16:07.349 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:07.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430449 got 206158430450 for osd.3 2026-03-31T21:16:07.355 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:07.355 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:07.588 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:07.588 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:07.601 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2026-03-31T21:16:05.908114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6607,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20249,"num_bytes_recovered":166153392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":349536,"kb_used_data":135976,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282765984,"statfs":{"total":289910292480,"available":289552367616,"internally_reserved":0,"allocated":139239424,"data_stored":138365570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-137,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001397"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":54064,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:04.724408+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:04.724408+0000","last_peered":"2026-03-31T21:16:04.724408+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:04.724408+0000","last_undegraded":"2026-03-31T21:16:04.724408+0000","last_fullsized":"2026-03-31T21:16:04.724408+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6607,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20247,"num_bytes_recovered":165825200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6607,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20247,"num_bytes_recovered":165825200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":191938560,"data_stored":191938560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430450,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266927,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313504,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757499,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75368,"kb_used_data":27800,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94296472,"statfs":{"total":96636764160,"available":96559587328,"internally_reserved":0,"allocated":28467200,"data_stored":28173526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":27693056,"data_stored":27693056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:07.602 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 54064, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:04.724408+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:04.724408+0000', 'last_peered': '2026-03-31T21:16:04.724408+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:04.724408+0000', 'last_undegraded': '2026-03-31T21:16:04.724408+0000', 'last_fullsized': '2026-03-31T21:16:04.724408+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6607, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20247, 'num_bytes_recovered': 165825200, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:07.602 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:12.603 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:12.603 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:12.603 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:12.603 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:12.715 INFO:teuthology.orchestra.run.vm00.stdout:206158430454 2026-03-31T21:16:12.715 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:12.718 INFO:teuthology.orchestra.run.vm00.stdout:90194313507 2026-03-31T21:16:12.718 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:12.719 INFO:teuthology.orchestra.run.vm00.stdout:163208757501 2026-03-31T21:16:12.720 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:12.726 INFO:teuthology.orchestra.run.vm00.stdout:227633266930 2026-03-31T21:16:12.726 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:12.938 INFO:teuthology.orchestra.run.vm00.stdout:90194313505 2026-03-31T21:16:12.942 INFO:teuthology.orchestra.run.vm00.stdout:227633266929 2026-03-31T21:16:12.946 INFO:teuthology.orchestra.run.vm00.stdout:163208757500 2026-03-31T21:16:12.946 INFO:teuthology.orchestra.run.vm00.stdout:206158430452 2026-03-31T21:16:12.956 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266930 got 227633266929 for osd.2 2026-03-31T21:16:12.957 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313507 got 90194313505 for osd.1 2026-03-31T21:16:12.959 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430454 got 206158430452 for osd.3 2026-03-31T21:16:12.961 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757501 got 163208757500 for osd.0 2026-03-31T21:16:13.957 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:13.957 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:13.959 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:13.961 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:14.190 INFO:teuthology.orchestra.run.vm00.stdout:206158430454 2026-03-31T21:16:14.191 INFO:teuthology.orchestra.run.vm00.stdout:90194313507 2026-03-31T21:16:14.203 INFO:teuthology.orchestra.run.vm00.stdout:227633266930 2026-03-31T21:16:14.204 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313507 got 90194313507 for osd.1 2026-03-31T21:16:14.204 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:14.207 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430454 got 206158430454 for osd.3 2026-03-31T21:16:14.207 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:14.217 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266930 got 227633266930 for osd.2 2026-03-31T21:16:14.218 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:14.251 INFO:teuthology.orchestra.run.vm00.stdout:163208757501 2026-03-31T21:16:14.264 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757501 got 163208757501 for osd.0 2026-03-31T21:16:14.264 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:14.264 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:14.506 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:14.506 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:14.518 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2026-03-31T21:16:13.909245+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6529,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20327,"num_bytes_recovered":166792368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":349852,"kb_used_data":136292,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282765668,"statfs":{"total":289910292480,"available":289552044032,"internally_reserved":0,"allocated":139563008,"data_stored":138689154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":54220,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:12.682562+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:12.682562+0000","last_peered":"2026-03-31T21:16:12.682562+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:12.682562+0000","last_undegraded":"2026-03-31T21:16:12.682562+0000","last_fullsized":"2026-03-31T21:16:12.682562+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6529,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20325,"num_bytes_recovered":166464176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6529,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20325,"num_bytes_recovered":166464176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192258048,"data_stored":192258048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430454,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266930,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313507,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757501,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75684,"kb_used_data":28116,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94296156,"statfs":{"total":96636764160,"available":96559263744,"internally_reserved":0,"allocated":28790784,"data_stored":28497110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28012544,"data_stored":28012544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:14.519 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 54220, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:12.682562+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:12.682562+0000', 'last_peered': '2026-03-31T21:16:12.682562+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:12.682562+0000', 'last_undegraded': '2026-03-31T21:16:12.682562+0000', 'last_fullsized': '2026-03-31T21:16:12.682562+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6529, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20325, 'num_bytes_recovered': 166464176, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:14.519 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:19.520 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:19.520 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:19.520 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:19.520 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:19.639 INFO:teuthology.orchestra.run.vm00.stdout:206158430456 2026-03-31T21:16:19.640 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:19.644 INFO:teuthology.orchestra.run.vm00.stdout:227633266932 2026-03-31T21:16:19.644 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:19.646 INFO:teuthology.orchestra.run.vm00.stdout:163208757503 2026-03-31T21:16:19.646 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:19.655 INFO:teuthology.orchestra.run.vm00.stdout:90194313509 2026-03-31T21:16:19.655 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:19.875 INFO:teuthology.orchestra.run.vm00.stdout:90194313508 2026-03-31T21:16:19.882 INFO:teuthology.orchestra.run.vm00.stdout:163208757502 2026-03-31T21:16:19.894 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313509 got 90194313508 for osd.1 2026-03-31T21:16:19.894 INFO:teuthology.orchestra.run.vm00.stdout:227633266931 2026-03-31T21:16:19.898 INFO:teuthology.orchestra.run.vm00.stdout:206158430455 2026-03-31T21:16:19.903 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757503 got 163208757502 for osd.0 2026-03-31T21:16:19.911 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266932 got 227633266931 for osd.2 2026-03-31T21:16:19.913 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430456 got 206158430455 for osd.3 2026-03-31T21:16:20.895 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:20.904 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:20.912 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:20.914 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:21.132 INFO:teuthology.orchestra.run.vm00.stdout:90194313509 2026-03-31T21:16:21.159 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313509 got 90194313509 for osd.1 2026-03-31T21:16:21.159 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:21.185 INFO:teuthology.orchestra.run.vm00.stdout:206158430457 2026-03-31T21:16:21.197 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430456 got 206158430457 for osd.3 2026-03-31T21:16:21.197 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:21.209 INFO:teuthology.orchestra.run.vm00.stdout:227633266932 2026-03-31T21:16:21.214 INFO:teuthology.orchestra.run.vm00.stdout:163208757504 2026-03-31T21:16:21.220 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266932 got 227633266932 for osd.2 2026-03-31T21:16:21.221 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:21.225 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757503 got 163208757504 for osd.0 2026-03-31T21:16:21.225 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:21.225 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:21.543 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:16:21.541+0000 7f6aeffff640 0 --2- 192.168.123.100:0/3525762074 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f6af007f8c0 0x7f6af0123130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:16:21.661 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:21.661 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:21.673 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2026-03-31T21:16:19.910123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6460,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20396,"num_bytes_recovered":167357616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":350128,"kb_used_data":136568,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282765392,"statfs":{"total":289910292480,"available":289551761408,"internally_reserved":0,"allocated":139845632,"data_stored":138971778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001724"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":54358,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:19.740409+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:19.740409+0000","last_peered":"2026-03-31T21:16:19.740409+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:19.740409+0000","last_undegraded":"2026-03-31T21:16:19.740409+0000","last_fullsized":"2026-03-31T21:16:19.740409+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6460,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20394,"num_bytes_recovered":167029424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6460,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20394,"num_bytes_recovered":167029424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192540672,"data_stored":192540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430457,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266932,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757504,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75960,"kb_used_data":28392,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94295880,"statfs":{"total":96636764160,"available":96558981120,"internally_reserved":0,"allocated":29073408,"data_stored":28779734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28295168,"data_stored":28295168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:21.673 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 54358, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:19.740409+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:19.740409+0000', 'last_peered': '2026-03-31T21:16:19.740409+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:19.740409+0000', 'last_undegraded': '2026-03-31T21:16:19.740409+0000', 'last_fullsized': '2026-03-31T21:16:19.740409+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6460, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20394, 'num_bytes_recovered': 167029424, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:21.673 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:26.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:26.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:26.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:26.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:26.783 INFO:teuthology.orchestra.run.vm00.stdout:90194313511 2026-03-31T21:16:26.784 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:26.789 INFO:teuthology.orchestra.run.vm00.stdout:206158430459 2026-03-31T21:16:26.790 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:26.791 INFO:teuthology.orchestra.run.vm00.stdout:163208757506 2026-03-31T21:16:26.791 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:26.813 INFO:teuthology.orchestra.run.vm00.stdout:227633266935 2026-03-31T21:16:26.813 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:26.998 INFO:teuthology.orchestra.run.vm00.stdout:90194313510 2026-03-31T21:16:27.017 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313511 got 90194313510 for osd.1 2026-03-31T21:16:27.029 INFO:teuthology.orchestra.run.vm00.stdout:206158430458 2026-03-31T21:16:27.029 INFO:teuthology.orchestra.run.vm00.stdout:163208757505 2026-03-31T21:16:27.043 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757506 got 163208757505 for osd.0 2026-03-31T21:16:27.043 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430459 got 206158430458 for osd.3 2026-03-31T21:16:27.088 INFO:teuthology.orchestra.run.vm00.stdout:227633266933 2026-03-31T21:16:27.102 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266935 got 227633266933 for osd.2 2026-03-31T21:16:28.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:28.044 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:28.044 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:28.103 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:28.277 INFO:teuthology.orchestra.run.vm00.stdout:90194313510 2026-03-31T21:16:28.296 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313511 got 90194313510 for osd.1 2026-03-31T21:16:28.301 INFO:teuthology.orchestra.run.vm00.stdout:163208757506 2026-03-31T21:16:28.318 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757506 got 163208757506 for osd.0 2026-03-31T21:16:28.318 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:28.347 INFO:teuthology.orchestra.run.vm00.stdout:206158430459 2026-03-31T21:16:28.362 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430459 got 206158430459 for osd.3 2026-03-31T21:16:28.362 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:28.407 INFO:teuthology.orchestra.run.vm00.stdout:227633266935 2026-03-31T21:16:28.423 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266935 got 227633266935 for osd.2 2026-03-31T21:16:28.423 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:29.296 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:29.528 INFO:teuthology.orchestra.run.vm00.stdout:90194313512 2026-03-31T21:16:29.543 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313511 got 90194313512 for osd.1 2026-03-31T21:16:29.543 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:29.543 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:29.787 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:29.787 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:29.803 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2026-03-31T21:16:27.911048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6396,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20460,"num_bytes_recovered":167881904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":350384,"kb_used_data":136824,"kb_used_omap":58,"kb_used_meta":213445,"kb_avail":282765136,"statfs":{"total":289910292480,"available":289551499264,"internally_reserved":0,"allocated":140107776,"data_stored":139233922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":218568177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-113,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001496"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":54486,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:26.679839+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:26.679839+0000","last_peered":"2026-03-31T21:16:26.679839+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:26.679839+0000","last_undegraded":"2026-03-31T21:16:26.679839+0000","last_fullsized":"2026-03-31T21:16:26.679839+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6396,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20458,"num_bytes_recovered":167553712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6396,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20458,"num_bytes_recovered":167553712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192802816,"data_stored":192802816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430459,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266935,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313512,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757506,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76216,"kb_used_data":28648,"kb_used_omap":17,"kb_used_meta":47534,"kb_avail":94295624,"statfs":{"total":96636764160,"available":96558718976,"internally_reserved":0,"allocated":29335552,"data_stored":29041878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":48675148},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28557312,"data_stored":28557312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:29.803 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 54486, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:26.679839+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:26.679839+0000', 'last_peered': '2026-03-31T21:16:26.679839+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:26.679839+0000', 'last_undegraded': '2026-03-31T21:16:26.679839+0000', 'last_fullsized': '2026-03-31T21:16:26.679839+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6396, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20458, 'num_bytes_recovered': 167553712, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:29.803 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:34.804 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:34.804 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:34.804 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:34.804 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:34.911 INFO:teuthology.orchestra.run.vm00.stdout:163208757509 2026-03-31T21:16:34.911 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:34.911 INFO:teuthology.orchestra.run.vm00.stdout:206158430462 2026-03-31T21:16:34.912 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:34.917 INFO:teuthology.orchestra.run.vm00.stdout:227633266937 2026-03-31T21:16:34.917 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:34.920 INFO:teuthology.orchestra.run.vm00.stdout:90194313514 2026-03-31T21:16:34.920 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:35.128 INFO:teuthology.orchestra.run.vm00.stdout:163208757507 2026-03-31T21:16:35.139 INFO:teuthology.orchestra.run.vm00.stdout:227633266936 2026-03-31T21:16:35.145 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757509 got 163208757507 for osd.0 2026-03-31T21:16:35.145 INFO:teuthology.orchestra.run.vm00.stdout:206158430460 2026-03-31T21:16:35.157 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430462 got 206158430460 for osd.3 2026-03-31T21:16:35.158 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266937 got 227633266936 for osd.2 2026-03-31T21:16:35.199 INFO:teuthology.orchestra.run.vm00.stdout:90194313513 2026-03-31T21:16:35.212 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313514 got 90194313513 for osd.1 2026-03-31T21:16:36.145 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:36.157 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:36.158 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:36.212 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:36.373 INFO:teuthology.orchestra.run.vm00.stdout:163208757509 2026-03-31T21:16:36.395 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757509 got 163208757509 for osd.0 2026-03-31T21:16:36.395 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:36.414 INFO:teuthology.orchestra.run.vm00.stdout:227633266937 2026-03-31T21:16:36.428 INFO:teuthology.orchestra.run.vm00.stdout:206158430462 2026-03-31T21:16:36.429 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266937 got 227633266937 for osd.2 2026-03-31T21:16:36.429 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:36.443 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430462 got 206158430462 for osd.3 2026-03-31T21:16:36.443 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:36.510 INFO:teuthology.orchestra.run.vm00.stdout:90194313514 2026-03-31T21:16:36.522 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313514 got 90194313514 for osd.1 2026-03-31T21:16:36.522 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:36.522 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:36.712 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:36.712 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:36.724 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2026-03-31T21:16:35.912151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6317,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20539,"num_bytes_recovered":168529072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":368740,"kb_used_data":137132,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282746780,"statfs":{"total":289910292480,"available":289532702720,"internally_reserved":0,"allocated":140423168,"data_stored":139549314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-143,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143,"num_bytes_recovered":1171456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":54644,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:34.800295+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:34.800295+0000","last_peered":"2026-03-31T21:16:34.800295+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:34.800295+0000","last_undegraded":"2026-03-31T21:16:34.800295+0000","last_fullsized":"2026-03-31T21:16:34.800295+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6317,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20537,"num_bytes_recovered":168200880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6317,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20537,"num_bytes_recovered":168200880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":193126400,"data_stored":193126400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430462,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266937,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313514,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94572,"kb_used_data":28956,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94277268,"statfs":{"total":96636764160,"available":96539922432,"internally_reserved":0,"allocated":29650944,"data_stored":29357270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28880896,"data_stored":28880896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:36.724 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 54644, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:34.800295+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:34.800295+0000', 'last_peered': '2026-03-31T21:16:34.800295+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:34.800295+0000', 'last_undegraded': '2026-03-31T21:16:34.800295+0000', 'last_fullsized': '2026-03-31T21:16:34.800295+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6317, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20537, 'num_bytes_recovered': 168200880, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:36.724 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:41.725 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:41.726 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:41.726 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:41.726 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:41.830 INFO:teuthology.orchestra.run.vm00.stdout:206158430464 2026-03-31T21:16:41.830 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:41.831 INFO:teuthology.orchestra.run.vm00.stdout:90194313516 2026-03-31T21:16:41.832 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:41.832 INFO:teuthology.orchestra.run.vm00.stdout:227633266940 2026-03-31T21:16:41.832 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:41.840 INFO:teuthology.orchestra.run.vm00.stdout:163208757511 2026-03-31T21:16:41.841 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:42.054 INFO:teuthology.orchestra.run.vm00.stdout:163208757511 2026-03-31T21:16:42.071 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757511 got 163208757511 for osd.0 2026-03-31T21:16:42.071 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:42.078 INFO:teuthology.orchestra.run.vm00.stdout:206158430464 2026-03-31T21:16:42.093 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430464 got 206158430464 for osd.3 2026-03-31T21:16:42.093 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:42.099 INFO:teuthology.orchestra.run.vm00.stdout:90194313516 2026-03-31T21:16:42.108 INFO:teuthology.orchestra.run.vm00.stdout:227633266940 2026-03-31T21:16:42.111 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313516 got 90194313516 for osd.1 2026-03-31T21:16:42.111 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:42.119 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266940 got 227633266940 for osd.2 2026-03-31T21:16:42.119 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:42.119 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:42.348 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:42.348 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:42.360 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2026-03-31T21:16:41.912991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6249,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20607,"num_bytes_recovered":169086128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":369012,"kb_used_data":137404,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282746508,"statfs":{"total":289910292480,"available":289532424192,"internally_reserved":0,"allocated":140701696,"data_stored":139827842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-117,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001646"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":54780,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:41.801119+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:41.801119+0000","last_peered":"2026-03-31T21:16:41.801119+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:41.801119+0000","last_undegraded":"2026-03-31T21:16:41.801119+0000","last_fullsized":"2026-03-31T21:16:41.801119+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6249,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20605,"num_bytes_recovered":168757936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6249,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20605,"num_bytes_recovered":168757936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":193404928,"data_stored":193404928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430464,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266940,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94844,"kb_used_data":29228,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94276996,"statfs":{"total":96636764160,"available":96539643904,"internally_reserved":0,"allocated":29929472,"data_stored":29635798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29159424,"data_stored":29159424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:42.360 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 54780, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:41.801119+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:41.801119+0000', 'last_peered': '2026-03-31T21:16:41.801119+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:41.801119+0000', 'last_undegraded': '2026-03-31T21:16:41.801119+0000', 'last_fullsized': '2026-03-31T21:16:41.801119+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6249, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20605, 'num_bytes_recovered': 168757936, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:42.360 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:47.361 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:47.361 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:47.361 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:47.361 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:47.485 INFO:teuthology.orchestra.run.vm00.stdout:90194313518 2026-03-31T21:16:47.486 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:47.486 INFO:teuthology.orchestra.run.vm00.stdout:206158430466 2026-03-31T21:16:47.486 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:47.487 INFO:teuthology.orchestra.run.vm00.stdout:163208757513 2026-03-31T21:16:47.487 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:47.496 INFO:teuthology.orchestra.run.vm00.stdout:227633266942 2026-03-31T21:16:47.496 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:47.744 INFO:teuthology.orchestra.run.vm00.stdout:90194313517 2026-03-31T21:16:47.744 INFO:teuthology.orchestra.run.vm00.stdout:206158430465 2026-03-31T21:16:47.758 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430466 got 206158430465 for osd.3 2026-03-31T21:16:47.761 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313518 got 90194313517 for osd.1 2026-03-31T21:16:47.763 INFO:teuthology.orchestra.run.vm00.stdout:163208757512 2026-03-31T21:16:47.775 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757513 got 163208757512 for osd.0 2026-03-31T21:16:47.804 INFO:teuthology.orchestra.run.vm00.stdout:227633266940 2026-03-31T21:16:47.816 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266942 got 227633266940 for osd.2 2026-03-31T21:16:48.758 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:48.762 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:48.776 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:48.816 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:49.029 INFO:teuthology.orchestra.run.vm00.stdout:90194313519 2026-03-31T21:16:49.036 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:16:49.034+0000 7fe8da215640 0 --2- 192.168.123.100:0/1903348646 >> v2:192.168.123.100:3300/0 conn(0x7fe8d407ede0 0x7fe8d407f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:16:49.036 INFO:teuthology.orchestra.run.vm00.stdout:163208757513 2026-03-31T21:16:49.051 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757513 got 163208757513 for osd.0 2026-03-31T21:16:49.051 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:49.051 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313518 got 90194313519 for osd.1 2026-03-31T21:16:49.051 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:49.087 INFO:teuthology.orchestra.run.vm00.stdout:227633266942 2026-03-31T21:16:49.103 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266942 got 227633266942 for osd.2 2026-03-31T21:16:49.104 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:49.176 INFO:teuthology.orchestra.run.vm00.stdout:206158430466 2026-03-31T21:16:49.189 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430466 got 206158430466 for osd.3 2026-03-31T21:16:49.189 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:49.189 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:49.401 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:49.401 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:49.413 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2026-03-31T21:16:47.913770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6195,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20661,"num_bytes_recovered":169528496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":369240,"kb_used_data":137632,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282746280,"statfs":{"total":289910292480,"available":289532190720,"internally_reserved":0,"allocated":140935168,"data_stored":140061314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":54888,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:47.409466+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:47.409466+0000","last_peered":"2026-03-31T21:16:47.409466+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:47.409466+0000","last_undegraded":"2026-03-31T21:16:47.409466+0000","last_fullsized":"2026-03-31T21:16:47.409466+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6195,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20659,"num_bytes_recovered":169200304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6195,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20659,"num_bytes_recovered":169200304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":193626112,"data_stored":193626112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430466,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266942,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757513,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95072,"kb_used_data":29456,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94276768,"statfs":{"total":96636764160,"available":96539410432,"internally_reserved":0,"allocated":30162944,"data_stored":29869270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29380608,"data_stored":29380608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:49.413 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 54888, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:47.409466+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:47.409466+0000', 'last_peered': '2026-03-31T21:16:47.409466+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:47.409466+0000', 'last_undegraded': '2026-03-31T21:16:47.409466+0000', 'last_fullsized': '2026-03-31T21:16:47.409466+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6195, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20659, 'num_bytes_recovered': 169200304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:49.413 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:16:54.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:16:54.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:16:54.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:16:54.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:16:54.534 INFO:teuthology.orchestra.run.vm00.stdout:206158430468 2026-03-31T21:16:54.535 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:54.546 INFO:teuthology.orchestra.run.vm00.stdout:90194313521 2026-03-31T21:16:54.547 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:54.554 INFO:teuthology.orchestra.run.vm00.stdout:163208757515 2026-03-31T21:16:54.554 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:54.558 INFO:teuthology.orchestra.run.vm00.stdout:227633266944 2026-03-31T21:16:54.558 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:54.791 INFO:teuthology.orchestra.run.vm00.stdout:90194313520 2026-03-31T21:16:54.791 INFO:teuthology.orchestra.run.vm00.stdout:206158430467 2026-03-31T21:16:54.794 INFO:teuthology.orchestra.run.vm00.stdout:163208757514 2026-03-31T21:16:54.805 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430468 got 206158430467 for osd.3 2026-03-31T21:16:54.807 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313521 got 90194313520 for osd.1 2026-03-31T21:16:54.810 INFO:teuthology.orchestra.run.vm00.stdout:227633266943 2026-03-31T21:16:54.814 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757515 got 163208757514 for osd.0 2026-03-31T21:16:54.822 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266944 got 227633266943 for osd.2 2026-03-31T21:16:55.806 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:55.808 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:55.814 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:55.823 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:56.041 INFO:teuthology.orchestra.run.vm00.stdout:206158430467 2026-03-31T21:16:56.047 INFO:teuthology.orchestra.run.vm00.stdout:90194313520 2026-03-31T21:16:56.056 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430468 got 206158430467 for osd.3 2026-03-31T21:16:56.068 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313521 got 90194313520 for osd.1 2026-03-31T21:16:56.089 INFO:teuthology.orchestra.run.vm00.stdout:227633266943 2026-03-31T21:16:56.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266944 got 227633266943 for osd.2 2026-03-31T21:16:56.104 INFO:teuthology.orchestra.run.vm00.stdout:163208757514 2026-03-31T21:16:56.123 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757515 got 163208757514 for osd.0 2026-03-31T21:16:57.057 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:16:57.069 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:16:57.105 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:16:57.124 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:16:57.287 INFO:teuthology.orchestra.run.vm00.stdout:206158430469 2026-03-31T21:16:57.307 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430468 got 206158430469 for osd.3 2026-03-31T21:16:57.307 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:57.336 INFO:teuthology.orchestra.run.vm00.stdout:90194313521 2026-03-31T21:16:57.353 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313521 got 90194313521 for osd.1 2026-03-31T21:16:57.353 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:57.385 INFO:teuthology.orchestra.run.vm00.stdout:227633266944 2026-03-31T21:16:57.386 INFO:teuthology.orchestra.run.vm00.stdout:163208757516 2026-03-31T21:16:57.397 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266944 got 227633266944 for osd.2 2026-03-31T21:16:57.397 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:57.403 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757515 got 163208757516 for osd.0 2026-03-31T21:16:57.403 DEBUG:teuthology.parallel:result is None 2026-03-31T21:16:57.403 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:16:57.660 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:16:57.660 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:16:57.674 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2026-03-31T21:16:55.914863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6127,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20729,"num_bytes_recovered":170085552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":369532,"kb_used_data":137924,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282745988,"statfs":{"total":289910292480,"available":289531891712,"internally_reserved":0,"allocated":141234176,"data_stored":140360322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001628"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55024,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:16:54.760243+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:16:54.760243+0000","last_peered":"2026-03-31T21:16:54.760243+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:16:54.760243+0000","last_undegraded":"2026-03-31T21:16:54.760243+0000","last_fullsized":"2026-03-31T21:16:54.760243+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6127,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20727,"num_bytes_recovered":169757360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6127,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20727,"num_bytes_recovered":169757360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":193904640,"data_stored":193904640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430469,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266944,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95364,"kb_used_data":29748,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94276476,"statfs":{"total":96636764160,"available":96539111424,"internally_reserved":0,"allocated":30461952,"data_stored":30168278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29659136,"data_stored":29659136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:16:57.675 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55024, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:16:54.760243+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:16:54.760243+0000', 'last_peered': '2026-03-31T21:16:54.760243+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:16:54.760243+0000', 'last_undegraded': '2026-03-31T21:16:54.760243+0000', 'last_fullsized': '2026-03-31T21:16:54.760243+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6127, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20727, 'num_bytes_recovered': 169757360, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:16:57.675 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:02.676 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:02.676 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:02.676 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:02.676 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:02.803 INFO:teuthology.orchestra.run.vm00.stdout:163208757518 2026-03-31T21:17:02.803 INFO:teuthology.orchestra.run.vm00.stdout:227633266947 2026-03-31T21:17:02.803 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:02.804 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:02.809 INFO:teuthology.orchestra.run.vm00.stdout:90194313524 2026-03-31T21:17:02.809 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:02.826 INFO:teuthology.orchestra.run.vm00.stdout:206158430471 2026-03-31T21:17:02.826 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:03.045 INFO:teuthology.orchestra.run.vm00.stdout:227633266946 2026-03-31T21:17:03.054 INFO:teuthology.orchestra.run.vm00.stdout:90194313522 2026-03-31T21:17:03.056 INFO:teuthology.orchestra.run.vm00.stdout:163208757517 2026-03-31T21:17:03.061 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266947 got 227633266946 for osd.2 2026-03-31T21:17:03.069 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313524 got 90194313522 for osd.1 2026-03-31T21:17:03.070 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757518 got 163208757517 for osd.0 2026-03-31T21:17:03.120 INFO:teuthology.orchestra.run.vm00.stdout:206158430470 2026-03-31T21:17:03.132 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430471 got 206158430470 for osd.3 2026-03-31T21:17:04.062 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:04.070 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:04.071 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:04.133 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:04.308 INFO:teuthology.orchestra.run.vm00.stdout:227633266947 2026-03-31T21:17:04.329 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266947 got 227633266947 for osd.2 2026-03-31T21:17:04.329 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:04.345 INFO:teuthology.orchestra.run.vm00.stdout:163208757518 2026-03-31T21:17:04.358 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757518 got 163208757518 for osd.0 2026-03-31T21:17:04.358 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:04.363 INFO:teuthology.orchestra.run.vm00.stdout:90194313525 2026-03-31T21:17:04.377 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313524 got 90194313525 for osd.1 2026-03-31T21:17:04.377 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:04.390 INFO:teuthology.orchestra.run.vm00.stdout:206158430471 2026-03-31T21:17:04.402 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430471 got 206158430471 for osd.3 2026-03-31T21:17:04.402 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:04.402 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:04.650 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:04.650 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:04.667 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2026-03-31T21:17:03.915919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6049,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20807,"num_bytes_recovered":170724528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":369836,"kb_used_data":138228,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282745684,"statfs":{"total":289910292480,"available":289531580416,"internally_reserved":0,"allocated":141545472,"data_stored":140671618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-123,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55180,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:02.783059+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:02.783059+0000","last_peered":"2026-03-31T21:17:02.783059+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:02.783059+0000","last_undegraded":"2026-03-31T21:17:02.783059+0000","last_fullsized":"2026-03-31T21:17:02.783059+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6049,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20805,"num_bytes_recovered":170396336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6049,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20805,"num_bytes_recovered":170396336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194224128,"data_stored":194224128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430471,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266947,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313525,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95668,"kb_used_data":30052,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94276172,"statfs":{"total":96636764160,"available":96538800128,"internally_reserved":0,"allocated":30773248,"data_stored":30479574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":29978624,"data_stored":29978624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:04.667 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55180, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:02.783059+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:02.783059+0000', 'last_peered': '2026-03-31T21:17:02.783059+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:02.783059+0000', 'last_undegraded': '2026-03-31T21:17:02.783059+0000', 'last_fullsized': '2026-03-31T21:17:02.783059+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6049, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20805, 'num_bytes_recovered': 170396336, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:04.667 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:09.667 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:09.668 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:09.668 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:09.668 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:09.778 INFO:teuthology.orchestra.run.vm00.stdout:163208757520 2026-03-31T21:17:09.778 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:09.780 INFO:teuthology.orchestra.run.vm00.stdout:206158430473 2026-03-31T21:17:09.780 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:09.786 INFO:teuthology.orchestra.run.vm00.stdout:90194313527 2026-03-31T21:17:09.786 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:09.791 INFO:teuthology.orchestra.run.vm00.stdout:227633266949 2026-03-31T21:17:09.791 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:10.010 INFO:teuthology.orchestra.run.vm00.stdout:227633266948 2026-03-31T21:17:10.011 INFO:teuthology.orchestra.run.vm00.stdout:206158430472 2026-03-31T21:17:10.016 INFO:teuthology.orchestra.run.vm00.stdout:90194313526 2026-03-31T21:17:10.025 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430473 got 206158430472 for osd.3 2026-03-31T21:17:10.025 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266949 got 227633266948 for osd.2 2026-03-31T21:17:10.030 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313527 got 90194313526 for osd.1 2026-03-31T21:17:10.066 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:17:10.064+0000 7f322ffff640 0 --2- 192.168.123.100:0/584160337 >> [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] conn(0x7f323807dfc0 0x7f32381e0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:17:10.194 INFO:teuthology.orchestra.run.vm00.stdout:163208757519 2026-03-31T21:17:10.207 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757520 got 163208757519 for osd.0 2026-03-31T21:17:11.026 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:11.026 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:11.031 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:11.207 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:11.234 INFO:teuthology.orchestra.run.vm00.stdout:206158430474 2026-03-31T21:17:11.245 INFO:teuthology.orchestra.run.vm00.stdout:227633266949 2026-03-31T21:17:11.250 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430473 got 206158430474 for osd.3 2026-03-31T21:17:11.250 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:11.259 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266949 got 227633266949 for osd.2 2026-03-31T21:17:11.259 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:11.295 INFO:teuthology.orchestra.run.vm00.stdout:90194313527 2026-03-31T21:17:11.309 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313527 got 90194313527 for osd.1 2026-03-31T21:17:11.309 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:11.428 INFO:teuthology.orchestra.run.vm00.stdout:163208757521 2026-03-31T21:17:11.440 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757520 got 163208757521 for osd.0 2026-03-31T21:17:11.440 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:11.441 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:11.634 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:11.634 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:11.645 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2026-03-31T21:17:09.916733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5983,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20873,"num_bytes_recovered":171265200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":370100,"kb_used_data":138492,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282745420,"statfs":{"total":289910292480,"available":289531310080,"internally_reserved":0,"allocated":141815808,"data_stored":140941954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001605"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55312,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:09.771434+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:09.771434+0000","last_peered":"2026-03-31T21:17:09.771434+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:09.771434+0000","last_undegraded":"2026-03-31T21:17:09.771434+0000","last_fullsized":"2026-03-31T21:17:09.771434+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5983,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20871,"num_bytes_recovered":170937008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5983,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20871,"num_bytes_recovered":170937008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194494464,"data_stored":194494464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430474,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266949,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313527,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757521,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95932,"kb_used_data":30316,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94275908,"statfs":{"total":96636764160,"available":96538529792,"internally_reserved":0,"allocated":31043584,"data_stored":30749910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30248960,"data_stored":30248960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:11.646 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55312, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:09.771434+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:09.771434+0000', 'last_peered': '2026-03-31T21:17:09.771434+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:09.771434+0000', 'last_undegraded': '2026-03-31T21:17:09.771434+0000', 'last_fullsized': '2026-03-31T21:17:09.771434+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5983, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20871, 'num_bytes_recovered': 170937008, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:11.646 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:16.646 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:16.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:16.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:16.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:16.755 INFO:teuthology.orchestra.run.vm00.stdout:227633266952 2026-03-31T21:17:16.755 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:16.756 INFO:teuthology.orchestra.run.vm00.stdout:90194313529 2026-03-31T21:17:16.756 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:16.760 INFO:teuthology.orchestra.run.vm00.stdout:163208757523 2026-03-31T21:17:16.761 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:16.762 INFO:teuthology.orchestra.run.vm00.stdout:206158430476 2026-03-31T21:17:16.762 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:16.966 INFO:teuthology.orchestra.run.vm00.stdout:163208757522 2026-03-31T21:17:16.975 INFO:teuthology.orchestra.run.vm00.stdout:90194313528 2026-03-31T21:17:16.977 INFO:teuthology.orchestra.run.vm00.stdout:227633266950 2026-03-31T21:17:16.978 INFO:teuthology.orchestra.run.vm00.stdout:206158430475 2026-03-31T21:17:16.980 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757523 got 163208757522 for osd.0 2026-03-31T21:17:16.987 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313529 got 90194313528 for osd.1 2026-03-31T21:17:16.992 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266952 got 227633266950 for osd.2 2026-03-31T21:17:16.993 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430476 got 206158430475 for osd.3 2026-03-31T21:17:17.981 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:17.988 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:17.993 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:17.994 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:18.247 INFO:teuthology.orchestra.run.vm00.stdout:227633266950 2026-03-31T21:17:18.250 INFO:teuthology.orchestra.run.vm00.stdout:90194313528 2026-03-31T21:17:18.252 INFO:teuthology.orchestra.run.vm00.stdout:206158430475 2026-03-31T21:17:18.265 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266952 got 227633266950 for osd.2 2026-03-31T21:17:18.266 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430476 got 206158430475 for osd.3 2026-03-31T21:17:18.266 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313529 got 90194313528 for osd.1 2026-03-31T21:17:18.273 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:17:18.272+0000 7f845e016640 0 --2- 192.168.123.100:0/3110238367 >> [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] conn(0x7f8458174500 0x7f8458108ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:17:18.410 INFO:teuthology.orchestra.run.vm00.stdout:163208757523 2026-03-31T21:17:18.423 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757523 got 163208757523 for osd.0 2026-03-31T21:17:18.423 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:19.265 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:19.266 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:19.267 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:19.471 INFO:teuthology.orchestra.run.vm00.stdout:90194313530 2026-03-31T21:17:19.479 INFO:teuthology.orchestra.run.vm00.stdout:227633266952 2026-03-31T21:17:19.482 INFO:teuthology.orchestra.run.vm00.stdout:206158430476 2026-03-31T21:17:19.485 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313529 got 90194313530 for osd.1 2026-03-31T21:17:19.485 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:19.492 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266952 got 227633266952 for osd.2 2026-03-31T21:17:19.492 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:19.495 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430476 got 206158430476 for osd.3 2026-03-31T21:17:19.495 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:19.495 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:19.724 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:19.724 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:19.735 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":542,"stamp":"2026-03-31T21:17:17.917724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5918,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20938,"num_bytes_recovered":171797680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":370360,"kb_used_data":138752,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282745160,"statfs":{"total":289910292480,"available":289531043840,"internally_reserved":0,"allocated":142082048,"data_stored":141208194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-112,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001491"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55442,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:16.737456+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:16.737456+0000","last_peered":"2026-03-31T21:17:16.737456+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:16.737456+0000","last_undegraded":"2026-03-31T21:17:16.737456+0000","last_fullsized":"2026-03-31T21:17:16.737456+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5918,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20936,"num_bytes_recovered":171469488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5918,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20936,"num_bytes_recovered":171469488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":194760704,"data_stored":194760704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430476,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266952,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757523,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96192,"kb_used_data":30576,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94275648,"statfs":{"total":96636764160,"available":96538263552,"internally_reserved":0,"allocated":31309824,"data_stored":31016150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30515200,"data_stored":30515200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:19.735 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55442, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:16.737456+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:16.737456+0000', 'last_peered': '2026-03-31T21:17:16.737456+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:16.737456+0000', 'last_undegraded': '2026-03-31T21:17:16.737456+0000', 'last_fullsized': '2026-03-31T21:17:16.737456+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5918, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 20936, 'num_bytes_recovered': 171469488, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:19.735 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:24.736 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:24.736 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:24.736 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:24.736 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:24.842 INFO:teuthology.orchestra.run.vm00.stdout:90194313532 2026-03-31T21:17:24.842 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:24.845 INFO:teuthology.orchestra.run.vm00.stdout:227633266954 2026-03-31T21:17:24.845 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:24.852 INFO:teuthology.orchestra.run.vm00.stdout:206158430479 2026-03-31T21:17:24.852 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:25.042 INFO:teuthology.orchestra.run.vm00.stdout:163208757526 2026-03-31T21:17:25.042 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:25.052 INFO:teuthology.orchestra.run.vm00.stdout:90194313531 2026-03-31T21:17:25.071 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313532 got 90194313531 for osd.1 2026-03-31T21:17:25.103 INFO:teuthology.orchestra.run.vm00.stdout:206158430477 2026-03-31T21:17:25.110 INFO:teuthology.orchestra.run.vm00.stdout:227633266953 2026-03-31T21:17:25.115 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430479 got 206158430477 for osd.3 2026-03-31T21:17:25.123 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266954 got 227633266953 for osd.2 2026-03-31T21:17:25.251 INFO:teuthology.orchestra.run.vm00.stdout:163208757524 2026-03-31T21:17:25.263 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757526 got 163208757524 for osd.0 2026-03-31T21:17:26.072 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:26.116 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:26.124 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:26.264 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:26.279 INFO:teuthology.orchestra.run.vm00.stdout:90194313532 2026-03-31T21:17:26.293 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313532 got 90194313532 for osd.1 2026-03-31T21:17:26.293 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:26.320 INFO:teuthology.orchestra.run.vm00.stdout:206158430479 2026-03-31T21:17:26.334 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430479 got 206158430479 for osd.3 2026-03-31T21:17:26.334 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:26.365 INFO:teuthology.orchestra.run.vm00.stdout:227633266954 2026-03-31T21:17:26.380 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266954 got 227633266954 for osd.2 2026-03-31T21:17:26.380 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:26.471 INFO:teuthology.orchestra.run.vm00.stdout:163208757526 2026-03-31T21:17:26.483 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757526 got 163208757526 for osd.0 2026-03-31T21:17:26.483 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:26.483 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:26.672 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:26.672 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:26.684 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":546,"stamp":"2026-03-31T21:17:25.918783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5842,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21014,"num_bytes_recovered":172420272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":370664,"kb_used_data":139056,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282744856,"statfs":{"total":289910292480,"available":289530732544,"internally_reserved":0,"allocated":142393344,"data_stored":141519490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-141,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":1155072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001515"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55594,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:24.756194+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:24.756194+0000","last_peered":"2026-03-31T21:17:24.756194+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:24.756194+0000","last_undegraded":"2026-03-31T21:17:24.756194+0000","last_fullsized":"2026-03-31T21:17:24.756194+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5842,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21012,"num_bytes_recovered":172092080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5842,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21012,"num_bytes_recovered":172092080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":195080192,"data_stored":195080192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430479,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266954,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313532,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757526,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96496,"kb_used_data":30880,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94275344,"statfs":{"total":96636764160,"available":96537952256,"internally_reserved":0,"allocated":31621120,"data_stored":31327446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30834688,"data_stored":30834688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:26.684 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55594, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:24.756194+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:24.756194+0000', 'last_peered': '2026-03-31T21:17:24.756194+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:24.756194+0000', 'last_undegraded': '2026-03-31T21:17:24.756194+0000', 'last_fullsized': '2026-03-31T21:17:24.756194+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5842, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21012, 'num_bytes_recovered': 172092080, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:26.684 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:31.685 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:31.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:31.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:31.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:31.788 INFO:teuthology.orchestra.run.vm00.stdout:227633266957 2026-03-31T21:17:31.788 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:31.789 INFO:teuthology.orchestra.run.vm00.stdout:206158430481 2026-03-31T21:17:31.789 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:31.795 INFO:teuthology.orchestra.run.vm00.stdout:90194313534 2026-03-31T21:17:31.796 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:31.799 INFO:teuthology.orchestra.run.vm00.stdout:163208757528 2026-03-31T21:17:31.799 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:32.005 INFO:teuthology.orchestra.run.vm00.stdout:206158430480 2026-03-31T21:17:32.006 INFO:teuthology.orchestra.run.vm00.stdout:90194313533 2026-03-31T21:17:32.007 INFO:teuthology.orchestra.run.vm00.stdout:227633266955 2026-03-31T21:17:32.020 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430481 got 206158430480 for osd.3 2026-03-31T21:17:32.020 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313534 got 90194313533 for osd.1 2026-03-31T21:17:32.025 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266957 got 227633266955 for osd.2 2026-03-31T21:17:32.136 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:17:32.134+0000 7f6881f55640 0 --2- 192.168.123.100:0/2744745312 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f687c07dde0 0x7f687c126f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:17:32.261 INFO:teuthology.orchestra.run.vm00.stdout:163208757528 2026-03-31T21:17:32.272 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757528 got 163208757528 for osd.0 2026-03-31T21:17:32.272 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:33.021 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:33.021 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:33.026 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:33.238 INFO:teuthology.orchestra.run.vm00.stdout:90194313534 2026-03-31T21:17:33.249 INFO:teuthology.orchestra.run.vm00.stdout:206158430481 2026-03-31T21:17:33.252 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313534 got 90194313534 for osd.1 2026-03-31T21:17:33.252 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:33.263 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430481 got 206158430481 for osd.3 2026-03-31T21:17:33.263 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:33.357 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:17:33.355+0000 7f75d2ffd640 0 --2- 192.168.123.100:0/2867344680 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f75d407f480 0x7f75d4057a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:17:33.484 INFO:teuthology.orchestra.run.vm00.stdout:227633266957 2026-03-31T21:17:33.496 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266957 got 227633266957 for osd.2 2026-03-31T21:17:33.497 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:33.497 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:33.774 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:33.774 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:33.785 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":549,"stamp":"2026-03-31T21:17:31.919541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5776,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21080,"num_bytes_recovered":172960944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":370932,"kb_used_data":139324,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282744588,"statfs":{"total":289910292480,"available":289530458112,"internally_reserved":0,"allocated":142667776,"data_stored":141793922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-113,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001570"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55726,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:31.720469+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:31.720469+0000","last_peered":"2026-03-31T21:17:31.720469+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:31.720469+0000","last_undegraded":"2026-03-31T21:17:31.720469+0000","last_fullsized":"2026-03-31T21:17:31.720469+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21078,"num_bytes_recovered":172632752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":55,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:13:37.575754+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:13:37.575754+0000","last_peered":"2026-03-31T21:13:37.575754+0000","last_clean":"2026-03-31T21:13:37.575754+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:13:37.575754+0000","last_undegraded":"2026-03-31T21:13:37.575754+0000","last_fullsized":"2026-03-31T21:13:37.575754+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21078,"num_bytes_recovered":172632752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":195342336,"data_stored":195342336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430481,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266957,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313534,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757528,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96764,"kb_used_data":31148,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94275076,"statfs":{"total":96636764160,"available":96537677824,"internally_reserved":0,"allocated":31895552,"data_stored":31601878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31096832,"data_stored":31096832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:33.786 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55726, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:31.720469+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:31.720469+0000', 'last_peered': '2026-03-31T21:17:31.720469+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:31.720469+0000', 'last_undegraded': '2026-03-31T21:17:31.720469+0000', 'last_fullsized': '2026-03-31T21:17:31.720469+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5776, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21078, 'num_bytes_recovered': 172632752, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:33.786 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:38.787 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:38.787 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:38.787 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:38.787 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:38.897 INFO:teuthology.orchestra.run.vm00.stdout:90194313537 2026-03-31T21:17:38.897 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:38.903 INFO:teuthology.orchestra.run.vm00.stdout:227633266959 2026-03-31T21:17:38.903 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:38.909 INFO:teuthology.orchestra.run.vm00.stdout:163208757530 2026-03-31T21:17:38.909 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:38.915 INFO:teuthology.orchestra.run.vm00.stdout:206158430483 2026-03-31T21:17:38.915 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:39.127 INFO:teuthology.orchestra.run.vm00.stdout:227633266958 2026-03-31T21:17:39.130 INFO:teuthology.orchestra.run.vm00.stdout:90194313536 2026-03-31T21:17:39.140 INFO:teuthology.orchestra.run.vm00.stdout:163208757529 2026-03-31T21:17:39.142 INFO:teuthology.orchestra.run.vm00.stdout:206158430482 2026-03-31T21:17:39.147 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266959 got 227633266958 for osd.2 2026-03-31T21:17:39.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313537 got 90194313536 for osd.1 2026-03-31T21:17:39.155 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757530 got 163208757529 for osd.0 2026-03-31T21:17:39.158 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430483 got 206158430482 for osd.3 2026-03-31T21:17:40.149 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:40.149 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:40.156 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:40.159 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:40.362 INFO:teuthology.orchestra.run.vm00.stdout:227633266959 2026-03-31T21:17:40.364 INFO:teuthology.orchestra.run.vm00.stdout:90194313537 2026-03-31T21:17:40.377 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266959 got 227633266959 for osd.2 2026-03-31T21:17:40.377 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:40.380 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313537 got 90194313537 for osd.1 2026-03-31T21:17:40.380 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:40.398 INFO:teuthology.orchestra.run.vm00.stdout:206158430484 2026-03-31T21:17:40.408 INFO:teuthology.orchestra.run.vm00.stdout:163208757531 2026-03-31T21:17:40.410 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430483 got 206158430484 for osd.3 2026-03-31T21:17:40.411 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:40.420 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757530 got 163208757531 for osd.0 2026-03-31T21:17:40.420 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:40.420 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:40.648 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:40.648 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:40.659 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2026-03-31T21:17:39.920527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5698,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21158,"num_bytes_recovered":173599920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":371248,"kb_used_data":139640,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282744272,"statfs":{"total":289910292480,"available":289530134528,"internally_reserved":0,"allocated":142991360,"data_stored":142117506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001469"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55882,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:39.708772+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:39.708772+0000","last_peered":"2026-03-31T21:17:39.708772+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:39.708772+0000","last_undegraded":"2026-03-31T21:17:39.708772+0000","last_fullsized":"2026-03-31T21:17:39.708772+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5698,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21156,"num_bytes_recovered":173271728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5698,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21156,"num_bytes_recovered":173271728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":195661824,"data_stored":195661824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430484,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266959,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313537,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757531,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97080,"kb_used_data":31464,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94274760,"statfs":{"total":96636764160,"available":96537354240,"internally_reserved":0,"allocated":32219136,"data_stored":31925462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31416320,"data_stored":31416320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:40.659 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55882, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:39.708772+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:39.708772+0000', 'last_peered': '2026-03-31T21:17:39.708772+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:39.708772+0000', 'last_undegraded': '2026-03-31T21:17:39.708772+0000', 'last_fullsized': '2026-03-31T21:17:39.708772+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5698, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21156, 'num_bytes_recovered': 173271728, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:40.659 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:45.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:45.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:45.661 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:45.661 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:45.773 INFO:teuthology.orchestra.run.vm00.stdout:163208757533 2026-03-31T21:17:45.774 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:45.776 INFO:teuthology.orchestra.run.vm00.stdout:227633266961 2026-03-31T21:17:45.776 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:45.778 INFO:teuthology.orchestra.run.vm00.stdout:206158430486 2026-03-31T21:17:45.778 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:45.784 INFO:teuthology.orchestra.run.vm00.stdout:90194313539 2026-03-31T21:17:45.784 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:45.994 INFO:teuthology.orchestra.run.vm00.stdout:227633266960 2026-03-31T21:17:45.996 INFO:teuthology.orchestra.run.vm00.stdout:163208757531 2026-03-31T21:17:46.007 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266961 got 227633266960 for osd.2 2026-03-31T21:17:46.007 INFO:teuthology.orchestra.run.vm00.stdout:206158430484 2026-03-31T21:17:46.014 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757533 got 163208757531 for osd.0 2026-03-31T21:17:46.022 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430486 got 206158430484 for osd.3 2026-03-31T21:17:46.048 INFO:teuthology.orchestra.run.vm00.stdout:90194313538 2026-03-31T21:17:46.060 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313539 got 90194313538 for osd.1 2026-03-31T21:17:47.008 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:47.014 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:47.023 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:47.061 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:47.238 INFO:teuthology.orchestra.run.vm00.stdout:227633266961 2026-03-31T21:17:47.256 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266961 got 227633266961 for osd.2 2026-03-31T21:17:47.257 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:47.270 INFO:teuthology.orchestra.run.vm00.stdout:163208757533 2026-03-31T21:17:47.282 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757533 got 163208757533 for osd.0 2026-03-31T21:17:47.282 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:47.302 INFO:teuthology.orchestra.run.vm00.stdout:206158430486 2026-03-31T21:17:47.314 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430486 got 206158430486 for osd.3 2026-03-31T21:17:47.314 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:47.345 INFO:teuthology.orchestra.run.vm00.stdout:90194313539 2026-03-31T21:17:47.356 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313539 got 90194313539 for osd.1 2026-03-31T21:17:47.356 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:47.356 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:47.585 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:47.585 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:47.596 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":556,"stamp":"2026-03-31T21:17:45.921401+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5640,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21216,"num_bytes_recovered":174075056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":371480,"kb_used_data":139872,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282744040,"statfs":{"total":289910292480,"available":289529896960,"internally_reserved":0,"allocated":143228928,"data_stored":142355074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-136,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001606"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":55998,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:45.679225+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:45.679225+0000","last_peered":"2026-03-31T21:17:45.679225+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:45.679225+0000","last_undegraded":"2026-03-31T21:17:45.679225+0000","last_fullsized":"2026-03-31T21:17:45.679225+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5640,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21214,"num_bytes_recovered":173746864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5640,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21214,"num_bytes_recovered":173746864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":195899392,"data_stored":195899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430486,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266961,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757533,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97312,"kb_used_data":31696,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94274528,"statfs":{"total":96636764160,"available":96537116672,"internally_reserved":0,"allocated":32456704,"data_stored":32163030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31653888,"data_stored":31653888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:47.597 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 55998, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:45.679225+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:45.679225+0000', 'last_peered': '2026-03-31T21:17:45.679225+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:45.679225+0000', 'last_undegraded': '2026-03-31T21:17:45.679225+0000', 'last_fullsized': '2026-03-31T21:17:45.679225+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5640, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21214, 'num_bytes_recovered': 173746864, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:47.597 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:52.598 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:52.598 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:52.599 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:52.599 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:52.705 INFO:teuthology.orchestra.run.vm00.stdout:206158430488 2026-03-31T21:17:52.705 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:52.714 INFO:teuthology.orchestra.run.vm00.stdout:90194313541 2026-03-31T21:17:52.714 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:52.715 INFO:teuthology.orchestra.run.vm00.stdout:227633266964 2026-03-31T21:17:52.715 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:52.717 INFO:teuthology.orchestra.run.vm00.stdout:163208757535 2026-03-31T21:17:52.717 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:52.926 INFO:teuthology.orchestra.run.vm00.stdout:227633266963 2026-03-31T21:17:52.930 INFO:teuthology.orchestra.run.vm00.stdout:206158430487 2026-03-31T21:17:52.931 INFO:teuthology.orchestra.run.vm00.stdout:163208757534 2026-03-31T21:17:52.939 INFO:teuthology.orchestra.run.vm00.stdout:90194313540 2026-03-31T21:17:52.942 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266964 got 227633266963 for osd.2 2026-03-31T21:17:52.943 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430488 got 206158430487 for osd.3 2026-03-31T21:17:52.948 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757535 got 163208757534 for osd.0 2026-03-31T21:17:52.952 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313541 got 90194313540 for osd.1 2026-03-31T21:17:53.943 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:53.943 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:53.948 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:53.953 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:54.160 INFO:teuthology.orchestra.run.vm00.stdout:227633266964 2026-03-31T21:17:54.172 INFO:teuthology.orchestra.run.vm00.stdout:206158430488 2026-03-31T21:17:54.179 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266964 got 227633266964 for osd.2 2026-03-31T21:17:54.179 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:54.190 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430488 got 206158430488 for osd.3 2026-03-31T21:17:54.190 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:54.210 INFO:teuthology.orchestra.run.vm00.stdout:90194313542 2026-03-31T21:17:54.212 INFO:teuthology.orchestra.run.vm00.stdout:163208757535 2026-03-31T21:17:54.224 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313541 got 90194313542 for osd.1 2026-03-31T21:17:54.224 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:54.228 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757535 got 163208757535 for osd.0 2026-03-31T21:17:54.229 DEBUG:teuthology.parallel:result is None 2026-03-31T21:17:54.229 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:17:54.460 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:17:54.460 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:17:54.472 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":560,"stamp":"2026-03-31T21:17:53.922389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5572,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21284,"num_bytes_recovered":174632112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":371716,"kb_used_data":140108,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282743804,"statfs":{"total":289910292480,"available":289529655296,"internally_reserved":0,"allocated":143470592,"data_stored":142596738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":1032192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001582"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":56134,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:52.662303+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:52.662303+0000","last_peered":"2026-03-31T21:17:52.662303+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:52.662303+0000","last_undegraded":"2026-03-31T21:17:52.662303+0000","last_fullsized":"2026-03-31T21:17:52.662303+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5572,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21282,"num_bytes_recovered":174303920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5572,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21282,"num_bytes_recovered":174303920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196177920,"data_stored":196177920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430488,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266964,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313542,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757535,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97548,"kb_used_data":31932,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94274292,"statfs":{"total":96636764160,"available":96536875008,"internally_reserved":0,"allocated":32698368,"data_stored":32404694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":31932416,"data_stored":31932416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:17:54.472 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 56134, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:52.662303+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:52.662303+0000', 'last_peered': '2026-03-31T21:17:52.662303+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:52.662303+0000', 'last_undegraded': '2026-03-31T21:17:52.662303+0000', 'last_fullsized': '2026-03-31T21:17:52.662303+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5572, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21282, 'num_bytes_recovered': 174303920, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:17:54.472 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:17:59.473 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:17:59.474 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:17:59.474 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:17:59.474 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:17:59.584 INFO:teuthology.orchestra.run.vm00.stdout:227633266966 2026-03-31T21:17:59.585 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:17:59.593 INFO:teuthology.orchestra.run.vm00.stdout:163208757537 2026-03-31T21:17:59.593 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:17:59.596 INFO:teuthology.orchestra.run.vm00.stdout:206158430490 2026-03-31T21:17:59.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:17:59.596 INFO:teuthology.orchestra.run.vm00.stdout:90194313544 2026-03-31T21:17:59.596 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:17:59.827 INFO:teuthology.orchestra.run.vm00.stdout:227633266965 2026-03-31T21:17:59.841 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266966 got 227633266965 for osd.2 2026-03-31T21:17:59.852 INFO:teuthology.orchestra.run.vm00.stdout:163208757536 2026-03-31T21:17:59.863 INFO:teuthology.orchestra.run.vm00.stdout:90194313543 2026-03-31T21:17:59.871 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757537 got 163208757536 for osd.0 2026-03-31T21:17:59.877 INFO:teuthology.orchestra.run.vm00.stdout:206158430489 2026-03-31T21:17:59.881 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313544 got 90194313543 for osd.1 2026-03-31T21:17:59.890 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430490 got 206158430489 for osd.3 2026-03-31T21:18:00.842 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:00.871 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:00.882 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:00.890 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:01.058 INFO:teuthology.orchestra.run.vm00.stdout:227633266966 2026-03-31T21:18:01.074 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266966 got 227633266966 for osd.2 2026-03-31T21:18:01.075 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:01.093 INFO:teuthology.orchestra.run.vm00.stdout:163208757538 2026-03-31T21:18:01.097 INFO:teuthology.orchestra.run.vm00.stdout:90194313544 2026-03-31T21:18:01.109 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757537 got 163208757538 for osd.0 2026-03-31T21:18:01.109 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:01.111 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313544 got 90194313544 for osd.1 2026-03-31T21:18:01.111 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:01.133 INFO:teuthology.orchestra.run.vm00.stdout:206158430491 2026-03-31T21:18:01.144 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430490 got 206158430491 for osd.3 2026-03-31T21:18:01.144 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:01.144 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:01.372 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:01.373 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:01.384 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2026-03-31T21:17:59.923141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5503,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21353,"num_bytes_recovered":175197360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":372024,"kb_used_data":140416,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282743496,"statfs":{"total":289910292480,"available":289529339904,"internally_reserved":0,"allocated":143785984,"data_stored":142912130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-137,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001479"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":56273,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:17:59.775061+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:17:59.775061+0000","last_peered":"2026-03-31T21:17:59.775061+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:17:59.775061+0000","last_undegraded":"2026-03-31T21:17:59.775061+0000","last_fullsized":"2026-03-31T21:17:59.775061+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5503,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21351,"num_bytes_recovered":174869168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5503,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21351,"num_bytes_recovered":174869168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196460544,"data_stored":196460544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430491,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266966,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757538,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":97856,"kb_used_data":32240,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94273984,"statfs":{"total":96636764160,"available":96536559616,"internally_reserved":0,"allocated":33013760,"data_stored":32720086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32215040,"data_stored":32215040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:01.384 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 56273, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:17:59.775061+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:17:59.775061+0000', 'last_peered': '2026-03-31T21:17:59.775061+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:17:59.775061+0000', 'last_undegraded': '2026-03-31T21:17:59.775061+0000', 'last_fullsized': '2026-03-31T21:17:59.775061+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5503, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21351, 'num_bytes_recovered': 174869168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:01.384 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:06.385 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:06.385 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:06.385 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:06.386 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:06.496 INFO:teuthology.orchestra.run.vm00.stdout:90194313546 2026-03-31T21:18:06.497 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:06.499 INFO:teuthology.orchestra.run.vm00.stdout:206158430493 2026-03-31T21:18:06.500 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:06.502 INFO:teuthology.orchestra.run.vm00.stdout:227633266968 2026-03-31T21:18:06.502 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:06.509 INFO:teuthology.orchestra.run.vm00.stdout:163208757540 2026-03-31T21:18:06.509 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:06.726 INFO:teuthology.orchestra.run.vm00.stdout:206158430492 2026-03-31T21:18:06.740 INFO:teuthology.orchestra.run.vm00.stdout:227633266967 2026-03-31T21:18:06.742 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430493 got 206158430492 for osd.3 2026-03-31T21:18:06.745 INFO:teuthology.orchestra.run.vm00.stdout:90194313545 2026-03-31T21:18:06.759 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266968 got 227633266967 for osd.2 2026-03-31T21:18:06.759 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313546 got 90194313545 for osd.1 2026-03-31T21:18:06.770 INFO:teuthology.orchestra.run.vm00.stdout:163208757539 2026-03-31T21:18:06.783 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757540 got 163208757539 for osd.0 2026-03-31T21:18:07.742 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:07.759 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:07.759 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:07.784 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:07.971 INFO:teuthology.orchestra.run.vm00.stdout:206158430492 2026-03-31T21:18:07.988 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430493 got 206158430492 for osd.3 2026-03-31T21:18:08.009 INFO:teuthology.orchestra.run.vm00.stdout:90194313545 2026-03-31T21:18:08.020 INFO:teuthology.orchestra.run.vm00.stdout:163208757539 2026-03-31T21:18:08.020 INFO:teuthology.orchestra.run.vm00.stdout:227633266967 2026-03-31T21:18:08.023 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313546 got 90194313545 for osd.1 2026-03-31T21:18:08.033 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757540 got 163208757539 for osd.0 2026-03-31T21:18:08.035 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266968 got 227633266967 for osd.2 2026-03-31T21:18:08.989 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:09.023 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:09.033 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:09.036 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:09.219 INFO:teuthology.orchestra.run.vm00.stdout:206158430493 2026-03-31T21:18:09.237 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430493 got 206158430493 for osd.3 2026-03-31T21:18:09.237 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:09.255 INFO:teuthology.orchestra.run.vm00.stdout:90194313547 2026-03-31T21:18:09.269 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313546 got 90194313547 for osd.1 2026-03-31T21:18:09.269 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:09.292 INFO:teuthology.orchestra.run.vm00.stdout:163208757540 2026-03-31T21:18:09.303 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757540 got 163208757540 for osd.0 2026-03-31T21:18:09.303 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:09.312 INFO:teuthology.orchestra.run.vm00.stdout:227633266969 2026-03-31T21:18:09.325 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266968 got 227633266969 for osd.2 2026-03-31T21:18:09.325 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:09.325 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:09.584 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:09.584 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:09.596 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2026-03-31T21:18:07.924239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5438,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21418,"num_bytes_recovered":175729840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":372256,"kb_used_data":140648,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282743264,"statfs":{"total":289910292480,"available":289529102336,"internally_reserved":0,"allocated":144023552,"data_stored":143149698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-114,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001603"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":56403,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:06.437917+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:06.437917+0000","last_peered":"2026-03-31T21:18:06.437917+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:06.437917+0000","last_undegraded":"2026-03-31T21:18:06.437917+0000","last_fullsized":"2026-03-31T21:18:06.437917+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21416,"num_bytes_recovered":175401648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21416,"num_bytes_recovered":175401648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196726784,"data_stored":196726784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430493,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266969,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757540,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98088,"kb_used_data":32472,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94273752,"statfs":{"total":96636764160,"available":96536322048,"internally_reserved":0,"allocated":33251328,"data_stored":32957654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32481280,"data_stored":32481280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:09.596 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 56403, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:06.437917+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:06.437917+0000', 'last_peered': '2026-03-31T21:18:06.437917+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:06.437917+0000', 'last_undegraded': '2026-03-31T21:18:06.437917+0000', 'last_fullsized': '2026-03-31T21:18:06.437917+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5438, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21416, 'num_bytes_recovered': 175401648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:09.596 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:14.597 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:14.597 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:14.597 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:14.597 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:14.702 INFO:teuthology.orchestra.run.vm00.stdout:90194313549 2026-03-31T21:18:14.702 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:14.708 INFO:teuthology.orchestra.run.vm00.stdout:163208757542 2026-03-31T21:18:14.708 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:14.710 INFO:teuthology.orchestra.run.vm00.stdout:206158430495 2026-03-31T21:18:14.711 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:14.900 INFO:teuthology.orchestra.run.vm00.stdout:227633266971 2026-03-31T21:18:14.900 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:14.907 INFO:teuthology.orchestra.run.vm00.stdout:90194313548 2026-03-31T21:18:14.922 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313549 got 90194313548 for osd.1 2026-03-31T21:18:14.960 INFO:teuthology.orchestra.run.vm00.stdout:163208757541 2026-03-31T21:18:14.963 INFO:teuthology.orchestra.run.vm00.stdout:206158430494 2026-03-31T21:18:14.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757542 got 163208757541 for osd.0 2026-03-31T21:18:14.977 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430495 got 206158430494 for osd.3 2026-03-31T21:18:15.108 INFO:teuthology.orchestra.run.vm00.stdout:227633266970 2026-03-31T21:18:15.120 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266971 got 227633266970 for osd.2 2026-03-31T21:18:15.923 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:15.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:15.978 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:16.120 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:16.125 INFO:teuthology.orchestra.run.vm00.stdout:90194313549 2026-03-31T21:18:16.140 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313549 got 90194313549 for osd.1 2026-03-31T21:18:16.140 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:16.207 INFO:teuthology.orchestra.run.vm00.stdout:163208757543 2026-03-31T21:18:16.220 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757542 got 163208757543 for osd.0 2026-03-31T21:18:16.220 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:16.227 INFO:teuthology.orchestra.run.vm00.stdout:206158430496 2026-03-31T21:18:16.239 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430495 got 206158430496 for osd.3 2026-03-31T21:18:16.240 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:16.323 INFO:teuthology.orchestra.run.vm00.stdout:227633266971 2026-03-31T21:18:16.334 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266971 got 227633266971 for osd.2 2026-03-31T21:18:16.334 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:16.334 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:16.537 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:16.537 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:16.549 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2026-03-31T21:18:15.925428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5357,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21499,"num_bytes_recovered":176393392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":372608,"kb_used_data":141000,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282742912,"statfs":{"total":289910292480,"available":289528741888,"internally_reserved":0,"allocated":144384000,"data_stored":143510146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001717"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":56565,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:14.783814+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:14.783814+0000","last_peered":"2026-03-31T21:18:14.783814+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:14.783814+0000","last_undegraded":"2026-03-31T21:18:14.783814+0000","last_fullsized":"2026-03-31T21:18:14.783814+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5357,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21497,"num_bytes_recovered":176065200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5357,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21497,"num_bytes_recovered":176065200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":197058560,"data_stored":197058560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430496,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266971,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313549,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757543,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98440,"kb_used_data":32824,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94273400,"statfs":{"total":96636764160,"available":96535961600,"internally_reserved":0,"allocated":33611776,"data_stored":33318102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32813056,"data_stored":32813056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:16.549 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 56565, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:14.783814+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:14.783814+0000', 'last_peered': '2026-03-31T21:18:14.783814+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:14.783814+0000', 'last_undegraded': '2026-03-31T21:18:14.783814+0000', 'last_fullsized': '2026-03-31T21:18:14.783814+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5357, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21497, 'num_bytes_recovered': 176065200, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:16.549 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:21.550 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:21.550 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:21.550 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:21.551 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:21.667 INFO:teuthology.orchestra.run.vm00.stdout:227633266973 2026-03-31T21:18:21.667 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:21.671 INFO:teuthology.orchestra.run.vm00.stdout:206158430498 2026-03-31T21:18:21.671 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:21.674 INFO:teuthology.orchestra.run.vm00.stdout:90194313551 2026-03-31T21:18:21.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:21.676 INFO:teuthology.orchestra.run.vm00.stdout:163208757545 2026-03-31T21:18:21.676 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:21.886 INFO:teuthology.orchestra.run.vm00.stdout:90194313550 2026-03-31T21:18:21.894 INFO:teuthology.orchestra.run.vm00.stdout:163208757544 2026-03-31T21:18:21.894 INFO:teuthology.orchestra.run.vm00.stdout:227633266972 2026-03-31T21:18:21.904 INFO:teuthology.orchestra.run.vm00.stdout:206158430497 2026-03-31T21:18:21.907 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313551 got 90194313550 for osd.1 2026-03-31T21:18:21.910 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757545 got 163208757544 for osd.0 2026-03-31T21:18:21.916 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266973 got 227633266972 for osd.2 2026-03-31T21:18:21.917 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430498 got 206158430497 for osd.3 2026-03-31T21:18:22.909 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:22.910 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:22.917 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:22.918 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:23.112 INFO:teuthology.orchestra.run.vm00.stdout:163208757545 2026-03-31T21:18:23.126 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757545 got 163208757545 for osd.0 2026-03-31T21:18:23.126 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:23.159 INFO:teuthology.orchestra.run.vm00.stdout:206158430498 2026-03-31T21:18:23.161 INFO:teuthology.orchestra.run.vm00.stdout:227633266974 2026-03-31T21:18:23.171 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430498 got 206158430498 for osd.3 2026-03-31T21:18:23.171 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:23.175 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266973 got 227633266974 for osd.2 2026-03-31T21:18:23.175 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:23.178 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:18:23.177+0000 7fdc637fe640 0 --2- 192.168.123.100:0/1792588002 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7fdc6407ed30 0x7fdc6407f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:18:23.306 INFO:teuthology.orchestra.run.vm00.stdout:90194313551 2026-03-31T21:18:23.320 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313551 got 90194313551 for osd.1 2026-03-31T21:18:23.320 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:23.320 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:23.558 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:23.558 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:23.571 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":574,"stamp":"2026-03-31T21:18:21.926228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5293,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21563,"num_bytes_recovered":176917680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":372868,"kb_used_data":141260,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282742652,"statfs":{"total":289910292480,"available":289528475648,"internally_reserved":0,"allocated":144650240,"data_stored":143776386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-113,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001696"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":56693,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:21.547540+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:21.547540+0000","last_peered":"2026-03-31T21:18:21.547540+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:21.547540+0000","last_undegraded":"2026-03-31T21:18:21.547540+0000","last_fullsized":"2026-03-31T21:18:21.547540+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5293,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21561,"num_bytes_recovered":176589488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5293,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21561,"num_bytes_recovered":176589488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":197324800,"data_stored":197324800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430498,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266974,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98700,"kb_used_data":33084,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94273140,"statfs":{"total":96636764160,"available":96535695360,"internally_reserved":0,"allocated":33878016,"data_stored":33584342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33079296,"data_stored":33079296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:23.571 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 56693, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:21.547540+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:21.547540+0000', 'last_peered': '2026-03-31T21:18:21.547540+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:21.547540+0000', 'last_undegraded': '2026-03-31T21:18:21.547540+0000', 'last_fullsized': '2026-03-31T21:18:21.547540+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5293, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21561, 'num_bytes_recovered': 176589488, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:23.571 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:28.572 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:28.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:28.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:28.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:28.683 INFO:teuthology.orchestra.run.vm00.stdout:90194313554 2026-03-31T21:18:28.683 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:28.686 INFO:teuthology.orchestra.run.vm00.stdout:206158430500 2026-03-31T21:18:28.686 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:28.686 INFO:teuthology.orchestra.run.vm00.stdout:227633266976 2026-03-31T21:18:28.687 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:28.690 INFO:teuthology.orchestra.run.vm00.stdout:163208757547 2026-03-31T21:18:28.690 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:28.894 INFO:teuthology.orchestra.run.vm00.stdout:206158430499 2026-03-31T21:18:28.895 INFO:teuthology.orchestra.run.vm00.stdout:90194313553 2026-03-31T21:18:28.895 INFO:teuthology.orchestra.run.vm00.stdout:227633266975 2026-03-31T21:18:28.904 INFO:teuthology.orchestra.run.vm00.stdout:163208757546 2026-03-31T21:18:28.909 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313554 got 90194313553 for osd.1 2026-03-31T21:18:28.909 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266976 got 227633266975 for osd.2 2026-03-31T21:18:28.909 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430500 got 206158430499 for osd.3 2026-03-31T21:18:28.917 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757547 got 163208757546 for osd.0 2026-03-31T21:18:29.909 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:29.910 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:29.910 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:29.917 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:30.129 INFO:teuthology.orchestra.run.vm00.stdout:227633266976 2026-03-31T21:18:30.131 INFO:teuthology.orchestra.run.vm00.stdout:206158430501 2026-03-31T21:18:30.137 INFO:teuthology.orchestra.run.vm00.stdout:90194313554 2026-03-31T21:18:30.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430500 got 206158430501 for osd.3 2026-03-31T21:18:30.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266976 got 227633266976 for osd.2 2026-03-31T21:18:30.148 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:30.148 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:30.150 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313554 got 90194313554 for osd.1 2026-03-31T21:18:30.150 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:30.180 INFO:teuthology.orchestra.run.vm00.stdout:163208757548 2026-03-31T21:18:30.191 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757547 got 163208757548 for osd.0 2026-03-31T21:18:30.191 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:30.191 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:30.421 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:30.422 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:30.431 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2026-03-31T21:18:29.927296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5213,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21643,"num_bytes_recovered":177573040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":373184,"kb_used_data":141576,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282742336,"statfs":{"total":289910292480,"available":289528152064,"internally_reserved":0,"allocated":144973824,"data_stored":144099970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001617"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":56853,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:29.761942+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:29.761942+0000","last_peered":"2026-03-31T21:18:29.761942+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:29.761942+0000","last_undegraded":"2026-03-31T21:18:29.761942+0000","last_fullsized":"2026-03-31T21:18:29.761942+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5213,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21641,"num_bytes_recovered":177244848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5213,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21641,"num_bytes_recovered":177244848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":197648384,"data_stored":197648384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430501,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266976,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313554,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757548,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99016,"kb_used_data":33400,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94272824,"statfs":{"total":96636764160,"available":96535371776,"internally_reserved":0,"allocated":34201600,"data_stored":33907926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33402880,"data_stored":33402880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:30.432 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 56853, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:29.761942+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:29.761942+0000', 'last_peered': '2026-03-31T21:18:29.761942+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:29.761942+0000', 'last_undegraded': '2026-03-31T21:18:29.761942+0000', 'last_fullsized': '2026-03-31T21:18:29.761942+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5213, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21641, 'num_bytes_recovered': 177244848, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:30.432 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:35.433 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:35.433 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:35.433 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:35.433 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:35.531 INFO:teuthology.orchestra.run.vm00.stdout:163208757550 2026-03-31T21:18:35.531 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:35.534 INFO:teuthology.orchestra.run.vm00.stdout:227633266978 2026-03-31T21:18:35.534 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:35.535 INFO:teuthology.orchestra.run.vm00.stdout:90194313556 2026-03-31T21:18:35.535 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:35.707 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:18:35.706+0000 7f0e72665640 0 --2- 192.168.123.100:0/1699970411 >> [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] conn(0x7f0e5c003aa0 0x7f0e5c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:18:35.729 INFO:teuthology.orchestra.run.vm00.stdout:163208757548 2026-03-31T21:18:35.745 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757550 got 163208757548 for osd.0 2026-03-31T21:18:35.780 INFO:teuthology.orchestra.run.vm00.stdout:227633266977 2026-03-31T21:18:35.782 INFO:teuthology.orchestra.run.vm00.stdout:90194313555 2026-03-31T21:18:35.791 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266978 got 227633266977 for osd.2 2026-03-31T21:18:35.793 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313556 got 90194313555 for osd.1 2026-03-31T21:18:35.931 INFO:teuthology.orchestra.run.vm00.stdout:206158430503 2026-03-31T21:18:35.931 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:36.127 INFO:teuthology.orchestra.run.vm00.stdout:206158430503 2026-03-31T21:18:36.139 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430503 got 206158430503 for osd.3 2026-03-31T21:18:36.139 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:36.746 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:36.792 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:36.794 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:36.950 INFO:teuthology.orchestra.run.vm00.stdout:163208757550 2026-03-31T21:18:36.963 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757550 got 163208757550 for osd.0 2026-03-31T21:18:36.963 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:36.994 INFO:teuthology.orchestra.run.vm00.stdout:90194313556 2026-03-31T21:18:37.000 INFO:teuthology.orchestra.run.vm00.stdout:227633266978 2026-03-31T21:18:37.005 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313556 got 90194313556 for osd.1 2026-03-31T21:18:37.005 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:37.013 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266978 got 227633266978 for osd.2 2026-03-31T21:18:37.013 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:37.013 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:37.243 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:37.243 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:37.254 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2026-03-31T21:18:35.928046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5153,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21703,"num_bytes_recovered":178064560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":373380,"kb_used_data":141772,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282742140,"statfs":{"total":289910292480,"available":289527951360,"internally_reserved":0,"allocated":145174528,"data_stored":144300674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-140,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001539"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":56973,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:35.907469+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:35.907469+0000","last_peered":"2026-03-31T21:18:35.907469+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:35.907469+0000","last_undegraded":"2026-03-31T21:18:35.907469+0000","last_fullsized":"2026-03-31T21:18:35.907469+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5153,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21701,"num_bytes_recovered":177736368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5153,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21701,"num_bytes_recovered":177736368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":197877760,"data_stored":197877760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430503,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266978,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757550,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99212,"kb_used_data":33596,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94272628,"statfs":{"total":96636764160,"available":96535171072,"internally_reserved":0,"allocated":34402304,"data_stored":34108630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33632256,"data_stored":33632256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:37.255 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 56973, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:35.907469+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:35.907469+0000', 'last_peered': '2026-03-31T21:18:35.907469+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:35.907469+0000', 'last_undegraded': '2026-03-31T21:18:35.907469+0000', 'last_fullsized': '2026-03-31T21:18:35.907469+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5153, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21701, 'num_bytes_recovered': 177736368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:37.255 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:42.256 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:42.256 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:42.256 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:42.256 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:42.369 INFO:teuthology.orchestra.run.vm00.stdout:90194313558 2026-03-31T21:18:42.370 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:42.370 INFO:teuthology.orchestra.run.vm00.stdout:227633266981 2026-03-31T21:18:42.370 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:42.372 INFO:teuthology.orchestra.run.vm00.stdout:163208757552 2026-03-31T21:18:42.372 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:42.374 INFO:teuthology.orchestra.run.vm00.stdout:206158430505 2026-03-31T21:18:42.375 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:42.579 INFO:teuthology.orchestra.run.vm00.stdout:163208757551 2026-03-31T21:18:42.595 INFO:teuthology.orchestra.run.vm00.stdout:227633266980 2026-03-31T21:18:42.598 INFO:teuthology.orchestra.run.vm00.stdout:206158430504 2026-03-31T21:18:42.599 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757552 got 163208757551 for osd.0 2026-03-31T21:18:42.601 INFO:teuthology.orchestra.run.vm00.stdout:90194313557 2026-03-31T21:18:42.611 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266981 got 227633266980 for osd.2 2026-03-31T21:18:42.614 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313558 got 90194313557 for osd.1 2026-03-31T21:18:42.615 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430505 got 206158430504 for osd.3 2026-03-31T21:18:43.599 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:43.611 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:43.614 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:43.616 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:43.804 INFO:teuthology.orchestra.run.vm00.stdout:163208757551 2026-03-31T21:18:43.824 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757552 got 163208757551 for osd.0 2026-03-31T21:18:43.849 INFO:teuthology.orchestra.run.vm00.stdout:90194313557 2026-03-31T21:18:43.850 INFO:teuthology.orchestra.run.vm00.stdout:206158430504 2026-03-31T21:18:43.855 INFO:teuthology.orchestra.run.vm00.stdout:227633266980 2026-03-31T21:18:43.863 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313558 got 90194313557 for osd.1 2026-03-31T21:18:43.867 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430505 got 206158430504 for osd.3 2026-03-31T21:18:43.869 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266981 got 227633266980 for osd.2 2026-03-31T21:18:44.825 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:44.863 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:44.867 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:44.870 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:45.053 INFO:teuthology.orchestra.run.vm00.stdout:163208757552 2026-03-31T21:18:45.069 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757552 got 163208757552 for osd.0 2026-03-31T21:18:45.069 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:45.087 INFO:teuthology.orchestra.run.vm00.stdout:206158430505 2026-03-31T21:18:45.101 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430505 got 206158430505 for osd.3 2026-03-31T21:18:45.101 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:45.117 INFO:teuthology.orchestra.run.vm00.stdout:90194313559 2026-03-31T21:18:45.130 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313558 got 90194313559 for osd.1 2026-03-31T21:18:45.130 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:45.135 INFO:teuthology.orchestra.run.vm00.stdout:227633266981 2026-03-31T21:18:45.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266981 got 227633266981 for osd.2 2026-03-31T21:18:45.146 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:45.146 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:45.373 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:45.373 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:45.385 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2026-03-31T21:18:43.928961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5091,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21765,"num_bytes_recovered":178572464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":373644,"kb_used_data":142036,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282741876,"statfs":{"total":289910292480,"available":289527681024,"internally_reserved":0,"allocated":145444864,"data_stored":144571010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001409"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":57097,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:42.281627+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:42.281627+0000","last_peered":"2026-03-31T21:18:42.281627+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:42.281627+0000","last_undegraded":"2026-03-31T21:18:42.281627+0000","last_fullsized":"2026-03-31T21:18:42.281627+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5091,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21763,"num_bytes_recovered":178244272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5091,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21763,"num_bytes_recovered":178244272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":198148096,"data_stored":198148096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430505,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266981,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313559,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757552,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99476,"kb_used_data":33860,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94272364,"statfs":{"total":96636764160,"available":96534900736,"internally_reserved":0,"allocated":34672640,"data_stored":34378966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33902592,"data_stored":33902592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:45.385 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 57097, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:42.281627+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:42.281627+0000', 'last_peered': '2026-03-31T21:18:42.281627+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:42.281627+0000', 'last_undegraded': '2026-03-31T21:18:42.281627+0000', 'last_fullsized': '2026-03-31T21:18:42.281627+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5091, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21763, 'num_bytes_recovered': 178244272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:45.385 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:50.386 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:50.387 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:50.387 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:50.387 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:50.489 INFO:teuthology.orchestra.run.vm00.stdout:227633266983 2026-03-31T21:18:50.489 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:50.491 INFO:teuthology.orchestra.run.vm00.stdout:206158430508 2026-03-31T21:18:50.491 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:50.498 INFO:teuthology.orchestra.run.vm00.stdout:90194313562 2026-03-31T21:18:50.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:50.692 INFO:teuthology.orchestra.run.vm00.stdout:227633266982 2026-03-31T21:18:50.701 INFO:teuthology.orchestra.run.vm00.stdout:163208757555 2026-03-31T21:18:50.701 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:50.709 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266983 got 227633266982 for osd.2 2026-03-31T21:18:50.737 INFO:teuthology.orchestra.run.vm00.stdout:90194313560 2026-03-31T21:18:50.743 INFO:teuthology.orchestra.run.vm00.stdout:206158430507 2026-03-31T21:18:50.752 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313562 got 90194313560 for osd.1 2026-03-31T21:18:50.755 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430508 got 206158430507 for osd.3 2026-03-31T21:18:50.891 INFO:teuthology.orchestra.run.vm00.stdout:163208757554 2026-03-31T21:18:50.901 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757555 got 163208757554 for osd.0 2026-03-31T21:18:51.710 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:51.753 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:51.755 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:51.901 INFO:teuthology.orchestra.run.vm00.stdout:227633266982 2026-03-31T21:18:51.902 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:51.912 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266983 got 227633266982 for osd.2 2026-03-31T21:18:51.959 INFO:teuthology.orchestra.run.vm00.stdout:90194313560 2026-03-31T21:18:51.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313562 got 90194313560 for osd.1 2026-03-31T21:18:51.995 INFO:teuthology.orchestra.run.vm00.stdout:206158430507 2026-03-31T21:18:52.011 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430508 got 206158430507 for osd.3 2026-03-31T21:18:52.097 INFO:teuthology.orchestra.run.vm00.stdout:163208757554 2026-03-31T21:18:52.108 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757555 got 163208757554 for osd.0 2026-03-31T21:18:52.912 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:52.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:53.012 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:53.105 INFO:teuthology.orchestra.run.vm00.stdout:227633266984 2026-03-31T21:18:53.109 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:53.116 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266983 got 227633266984 for osd.2 2026-03-31T21:18:53.116 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:53.170 INFO:teuthology.orchestra.run.vm00.stdout:90194313562 2026-03-31T21:18:53.181 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313562 got 90194313562 for osd.1 2026-03-31T21:18:53.181 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:53.287 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:18:53.284+0000 7f9801b80640 0 --2- 192.168.123.100:0/2476917942 >> [v2:192.168.123.100:3301/0,v1:192.168.123.100:6790/0] conn(0x7f97fc1519a0 0x7f97fc171d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:18:53.310 INFO:teuthology.orchestra.run.vm00.stdout:163208757555 2026-03-31T21:18:53.323 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757555 got 163208757555 for osd.0 2026-03-31T21:18:53.323 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:53.414 INFO:teuthology.orchestra.run.vm00.stdout:206158430508 2026-03-31T21:18:53.426 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430508 got 206158430508 for osd.3 2026-03-31T21:18:53.426 DEBUG:teuthology.parallel:result is None 2026-03-31T21:18:53.426 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:18:53.608 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:18:53.608 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:18:53.618 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2026-03-31T21:18:51.929919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5012,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21844,"num_bytes_recovered":179219632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":373992,"kb_used_data":142384,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282741528,"statfs":{"total":289910292480,"available":289527324672,"internally_reserved":0,"allocated":145801216,"data_stored":144927362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001410"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":57255,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:50.398670+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:50.398670+0000","last_peered":"2026-03-31T21:18:50.398670+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:50.398670+0000","last_undegraded":"2026-03-31T21:18:50.398670+0000","last_fullsized":"2026-03-31T21:18:50.398670+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5012,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21842,"num_bytes_recovered":178891440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5012,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21842,"num_bytes_recovered":178891440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":198479872,"data_stored":198479872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430508,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313562,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757555,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":99824,"kb_used_data":34208,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94272016,"statfs":{"total":96636764160,"available":96534544384,"internally_reserved":0,"allocated":35028992,"data_stored":34735318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34234368,"data_stored":34234368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:18:53.619 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 57255, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:50.398670+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:50.398670+0000', 'last_peered': '2026-03-31T21:18:50.398670+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:50.398670+0000', 'last_undegraded': '2026-03-31T21:18:50.398670+0000', 'last_fullsized': '2026-03-31T21:18:50.398670+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5012, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21842, 'num_bytes_recovered': 178891440, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:18:53.619 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:18:58.620 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:18:58.620 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:18:58.620 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:18:58.620 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:18:58.721 INFO:teuthology.orchestra.run.vm00.stdout:90194313565 2026-03-31T21:18:58.722 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:58.722 INFO:teuthology.orchestra.run.vm00.stdout:227633266986 2026-03-31T21:18:58.722 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:58.725 INFO:teuthology.orchestra.run.vm00.stdout:206158430510 2026-03-31T21:18:58.725 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:58.731 INFO:teuthology.orchestra.run.vm00.stdout:163208757557 2026-03-31T21:18:58.732 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:18:58.928 INFO:teuthology.orchestra.run.vm00.stdout:90194313564 2026-03-31T21:18:58.928 INFO:teuthology.orchestra.run.vm00.stdout:227633266985 2026-03-31T21:18:58.941 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266986 got 227633266985 for osd.2 2026-03-31T21:18:58.941 INFO:teuthology.orchestra.run.vm00.stdout:206158430509 2026-03-31T21:18:58.941 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313565 got 90194313564 for osd.1 2026-03-31T21:18:58.952 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430510 got 206158430509 for osd.3 2026-03-31T21:18:58.984 INFO:teuthology.orchestra.run.vm00.stdout:163208757556 2026-03-31T21:18:58.994 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757557 got 163208757556 for osd.0 2026-03-31T21:18:59.942 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:18:59.942 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:18:59.952 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:18:59.995 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:00.143 INFO:teuthology.orchestra.run.vm00.stdout:90194313564 2026-03-31T21:19:00.148 INFO:teuthology.orchestra.run.vm00.stdout:227633266985 2026-03-31T21:19:00.157 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313565 got 90194313564 for osd.1 2026-03-31T21:19:00.168 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266986 got 227633266985 for osd.2 2026-03-31T21:19:00.193 INFO:teuthology.orchestra.run.vm00.stdout:206158430511 2026-03-31T21:19:00.203 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430510 got 206158430511 for osd.3 2026-03-31T21:19:00.203 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:00.251 INFO:teuthology.orchestra.run.vm00.stdout:163208757558 2026-03-31T21:19:00.261 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757557 got 163208757558 for osd.0 2026-03-31T21:19:00.262 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:01.158 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:01.169 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:01.353 INFO:teuthology.orchestra.run.vm00.stdout:90194313565 2026-03-31T21:19:01.364 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313565 got 90194313565 for osd.1 2026-03-31T21:19:01.365 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:01.391 INFO:teuthology.orchestra.run.vm00.stdout:227633266986 2026-03-31T21:19:01.403 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266986 got 227633266986 for osd.2 2026-03-31T21:19:01.403 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:01.403 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:01.625 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:01.626 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:01.636 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":593,"stamp":"2026-03-31T21:18:59.930863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4921,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21935,"num_bytes_recovered":179965104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":374344,"kb_used_data":142736,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282741176,"statfs":{"total":289910292480,"available":289526964224,"internally_reserved":0,"allocated":146161664,"data_stored":145287810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001403"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":57437,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:18:59.726290+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:18:59.726290+0000","last_peered":"2026-03-31T21:18:59.726290+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:18:59.726290+0000","last_undegraded":"2026-03-31T21:18:59.726290+0000","last_fullsized":"2026-03-31T21:18:59.726290+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4921,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21933,"num_bytes_recovered":179636912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4921,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21933,"num_bytes_recovered":179636912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":198844416,"data_stored":198844416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430511,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266986,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100176,"kb_used_data":34560,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94271664,"statfs":{"total":96636764160,"available":96534183936,"internally_reserved":0,"allocated":35389440,"data_stored":35095766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34598912,"data_stored":34598912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:01.636 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 57437, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:18:59.726290+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:18:59.726290+0000', 'last_peered': '2026-03-31T21:18:59.726290+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:18:59.726290+0000', 'last_undegraded': '2026-03-31T21:18:59.726290+0000', 'last_fullsized': '2026-03-31T21:18:59.726290+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4921, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 21933, 'num_bytes_recovered': 179636912, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:01.636 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:06.636 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:06.637 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:06.637 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:06.637 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:06.739 INFO:teuthology.orchestra.run.vm00.stdout:227633266989 2026-03-31T21:19:06.739 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:06.743 INFO:teuthology.orchestra.run.vm00.stdout:90194313567 2026-03-31T21:19:06.743 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:06.745 INFO:teuthology.orchestra.run.vm00.stdout:163208757560 2026-03-31T21:19:06.746 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:06.752 INFO:teuthology.orchestra.run.vm00.stdout:206158430513 2026-03-31T21:19:06.752 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:06.951 INFO:teuthology.orchestra.run.vm00.stdout:227633266987 2026-03-31T21:19:06.951 INFO:teuthology.orchestra.run.vm00.stdout:90194313566 2026-03-31T21:19:06.952 INFO:teuthology.orchestra.run.vm00.stdout:206158430512 2026-03-31T21:19:06.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430513 got 206158430512 for osd.3 2026-03-31T21:19:06.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313567 got 90194313566 for osd.1 2026-03-31T21:19:06.968 INFO:teuthology.orchestra.run.vm00.stdout:163208757559 2026-03-31T21:19:06.969 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266989 got 227633266987 for osd.2 2026-03-31T21:19:06.977 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757560 got 163208757559 for osd.0 2026-03-31T21:19:07.965 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:07.965 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:07.970 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:07.978 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:08.166 INFO:teuthology.orchestra.run.vm00.stdout:206158430512 2026-03-31T21:19:08.181 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430513 got 206158430512 for osd.3 2026-03-31T21:19:08.197 INFO:teuthology.orchestra.run.vm00.stdout:90194313568 2026-03-31T21:19:08.211 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313567 got 90194313568 for osd.1 2026-03-31T21:19:08.211 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:08.223 INFO:teuthology.orchestra.run.vm00.stdout:227633266989 2026-03-31T21:19:08.228 INFO:teuthology.orchestra.run.vm00.stdout:163208757560 2026-03-31T21:19:08.237 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266989 got 227633266989 for osd.2 2026-03-31T21:19:08.237 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:08.243 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757560 got 163208757560 for osd.0 2026-03-31T21:19:08.243 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:09.182 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:09.377 INFO:teuthology.orchestra.run.vm00.stdout:206158430513 2026-03-31T21:19:09.389 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430513 got 206158430513 for osd.3 2026-03-31T21:19:09.389 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:09.389 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:09.579 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:09.579 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:09.590 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":597,"stamp":"2026-03-31T21:19:07.931817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4853,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22003,"num_bytes_recovered":180522160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":374612,"kb_used_data":143004,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282740908,"statfs":{"total":289910292480,"available":289526689792,"internally_reserved":0,"allocated":146436096,"data_stored":145562242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-117,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":958464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001401"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":57573,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:19:06.684040+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:19:06.684040+0000","last_peered":"2026-03-31T21:19:06.684040+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:06.684040+0000","last_undegraded":"2026-03-31T21:19:06.684040+0000","last_fullsized":"2026-03-31T21:19:06.684040+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4853,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22001,"num_bytes_recovered":180193968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4853,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22001,"num_bytes_recovered":180193968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199122944,"data_stored":199122944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430513,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757560,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100444,"kb_used_data":34828,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94271396,"statfs":{"total":96636764160,"available":96533909504,"internally_reserved":0,"allocated":35663872,"data_stored":35370198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34877440,"data_stored":34877440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:09.590 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 57573, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:19:06.684040+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:19:06.684040+0000', 'last_peered': '2026-03-31T21:19:06.684040+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:06.684040+0000', 'last_undegraded': '2026-03-31T21:19:06.684040+0000', 'last_fullsized': '2026-03-31T21:19:06.684040+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4853, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22001, 'num_bytes_recovered': 180193968, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:09.590 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:14.591 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:14.591 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:14.592 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:14.592 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:14.692 INFO:teuthology.orchestra.run.vm00.stdout:90194313570 2026-03-31T21:19:14.692 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:14.695 INFO:teuthology.orchestra.run.vm00.stdout:206158430515 2026-03-31T21:19:14.695 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:14.702 INFO:teuthology.orchestra.run.vm00.stdout:227633266991 2026-03-31T21:19:14.702 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:14.702 INFO:teuthology.orchestra.run.vm00.stdout:163208757562 2026-03-31T21:19:14.703 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:14.903 INFO:teuthology.orchestra.run.vm00.stdout:227633266990 2026-03-31T21:19:14.906 INFO:teuthology.orchestra.run.vm00.stdout:90194313569 2026-03-31T21:19:14.906 INFO:teuthology.orchestra.run.vm00.stdout:163208757561 2026-03-31T21:19:14.909 INFO:teuthology.orchestra.run.vm00.stdout:206158430514 2026-03-31T21:19:14.919 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266991 got 227633266990 for osd.2 2026-03-31T21:19:14.919 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313570 got 90194313569 for osd.1 2026-03-31T21:19:14.922 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430515 got 206158430514 for osd.3 2026-03-31T21:19:14.923 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757562 got 163208757561 for osd.0 2026-03-31T21:19:15.920 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:15.920 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:15.922 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:15.923 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:16.128 INFO:teuthology.orchestra.run.vm00.stdout:206158430514 2026-03-31T21:19:16.133 INFO:teuthology.orchestra.run.vm00.stdout:90194313569 2026-03-31T21:19:16.133 INFO:teuthology.orchestra.run.vm00.stdout:227633266990 2026-03-31T21:19:16.147 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430515 got 206158430514 for osd.3 2026-03-31T21:19:16.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313570 got 90194313569 for osd.1 2026-03-31T21:19:16.150 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266991 got 227633266990 for osd.2 2026-03-31T21:19:16.176 INFO:teuthology.orchestra.run.vm00.stdout:163208757561 2026-03-31T21:19:16.187 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757562 got 163208757561 for osd.0 2026-03-31T21:19:17.148 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:17.148 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:17.150 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:17.188 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:17.356 INFO:teuthology.orchestra.run.vm00.stdout:206158430516 2026-03-31T21:19:17.364 INFO:teuthology.orchestra.run.vm00.stdout:90194313570 2026-03-31T21:19:17.373 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430515 got 206158430516 for osd.3 2026-03-31T21:19:17.373 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:17.380 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313570 got 90194313570 for osd.1 2026-03-31T21:19:17.380 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:17.401 INFO:teuthology.orchestra.run.vm00.stdout:163208757563 2026-03-31T21:19:17.413 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757562 got 163208757563 for osd.0 2026-03-31T21:19:17.413 INFO:teuthology.orchestra.run.vm00.stdout:227633266991 2026-03-31T21:19:17.413 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:17.425 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266991 got 227633266991 for osd.2 2026-03-31T21:19:17.425 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:17.425 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:17.649 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:17.649 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:17.659 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2026-03-31T21:19:15.932799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4774,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22082,"num_bytes_recovered":181169328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":374932,"kb_used_data":143324,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282740588,"statfs":{"total":289910292480,"available":289526362112,"internally_reserved":0,"allocated":146763776,"data_stored":145889922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001495"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":57731,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:19:14.763856+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:19:14.763856+0000","last_peered":"2026-03-31T21:19:14.763856+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:14.763856+0000","last_undegraded":"2026-03-31T21:19:14.763856+0000","last_fullsized":"2026-03-31T21:19:14.763856+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4774,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22080,"num_bytes_recovered":180841136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4774,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22080,"num_bytes_recovered":180841136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199446528,"data_stored":199446528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757563,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":100764,"kb_used_data":35148,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94271076,"statfs":{"total":96636764160,"available":96533581824,"internally_reserved":0,"allocated":35991552,"data_stored":35697878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35201024,"data_stored":35201024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:17.659 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 57731, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:19:14.763856+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:19:14.763856+0000', 'last_peered': '2026-03-31T21:19:14.763856+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:14.763856+0000', 'last_undegraded': '2026-03-31T21:19:14.763856+0000', 'last_fullsized': '2026-03-31T21:19:14.763856+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4774, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22080, 'num_bytes_recovered': 180841136, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:17.659 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:22.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:22.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:22.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:22.660 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:22.766 INFO:teuthology.orchestra.run.vm00.stdout:227633266994 2026-03-31T21:19:22.766 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:22.766 INFO:teuthology.orchestra.run.vm00.stdout:206158430518 2026-03-31T21:19:22.766 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:22.773 INFO:teuthology.orchestra.run.vm00.stdout:90194313572 2026-03-31T21:19:22.773 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:22.774 INFO:teuthology.orchestra.run.vm00.stdout:163208757565 2026-03-31T21:19:22.774 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:22.979 INFO:teuthology.orchestra.run.vm00.stdout:227633266993 2026-03-31T21:19:22.981 INFO:teuthology.orchestra.run.vm00.stdout:206158430517 2026-03-31T21:19:22.993 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430518 got 206158430517 for osd.3 2026-03-31T21:19:22.994 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266994 got 227633266993 for osd.2 2026-03-31T21:19:23.024 INFO:teuthology.orchestra.run.vm00.stdout:163208757564 2026-03-31T21:19:23.035 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757565 got 163208757564 for osd.0 2026-03-31T21:19:23.044 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:19:23.042+0000 7f36b1847640 0 --2- 192.168.123.100:0/1123836505 >> v2:192.168.123.100:3300/0 conn(0x7f36ac1734f0 0x7f36ac1938d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:19:23.169 INFO:teuthology.orchestra.run.vm00.stdout:90194313571 2026-03-31T21:19:23.179 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313572 got 90194313571 for osd.1 2026-03-31T21:19:23.994 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:23.994 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:24.036 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:24.180 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:24.190 INFO:teuthology.orchestra.run.vm00.stdout:227633266993 2026-03-31T21:19:24.196 INFO:teuthology.orchestra.run.vm00.stdout:206158430517 2026-03-31T21:19:24.203 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266994 got 227633266993 for osd.2 2026-03-31T21:19:24.212 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430518 got 206158430517 for osd.3 2026-03-31T21:19:24.239 INFO:teuthology.orchestra.run.vm00.stdout:163208757564 2026-03-31T21:19:24.254 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757565 got 163208757564 for osd.0 2026-03-31T21:19:24.375 INFO:teuthology.orchestra.run.vm00.stdout:90194313573 2026-03-31T21:19:24.385 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313572 got 90194313573 for osd.1 2026-03-31T21:19:24.385 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:25.204 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:25.213 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:25.254 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:25.398 INFO:teuthology.orchestra.run.vm00.stdout:227633266994 2026-03-31T21:19:25.410 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266994 got 227633266994 for osd.2 2026-03-31T21:19:25.411 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:25.440 INFO:teuthology.orchestra.run.vm00.stdout:206158430518 2026-03-31T21:19:25.451 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430518 got 206158430518 for osd.3 2026-03-31T21:19:25.451 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:25.489 INFO:teuthology.orchestra.run.vm00.stdout:163208757565 2026-03-31T21:19:25.499 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757565 got 163208757565 for osd.0 2026-03-31T21:19:25.499 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:25.499 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:25.721 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:25.721 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:25.731 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":605,"stamp":"2026-03-31T21:19:23.933823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4696,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22160,"num_bytes_recovered":181808304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":375240,"kb_used_data":143632,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282740280,"statfs":{"total":289910292480,"available":289526046720,"internally_reserved":0,"allocated":147079168,"data_stored":146205314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001466"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":57887,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:19:22.749900+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:19:22.749900+0000","last_peered":"2026-03-31T21:19:22.749900+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:22.749900+0000","last_undegraded":"2026-03-31T21:19:22.749900+0000","last_fullsized":"2026-03-31T21:19:22.749900+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4696,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22158,"num_bytes_recovered":181480112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4696,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22158,"num_bytes_recovered":181480112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199766016,"data_stored":199766016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101072,"kb_used_data":35456,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94270768,"statfs":{"total":96636764160,"available":96533266432,"internally_reserved":0,"allocated":36306944,"data_stored":36013270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35520512,"data_stored":35520512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:25.731 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 57887, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:19:22.749900+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:19:22.749900+0000', 'last_peered': '2026-03-31T21:19:22.749900+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:22.749900+0000', 'last_undegraded': '2026-03-31T21:19:22.749900+0000', 'last_fullsized': '2026-03-31T21:19:22.749900+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4696, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22158, 'num_bytes_recovered': 181480112, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:25.731 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:30.732 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:30.732 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:30.732 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:30.732 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:30.846 INFO:teuthology.orchestra.run.vm00.stdout:163208757568 2026-03-31T21:19:30.846 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:30.848 INFO:teuthology.orchestra.run.vm00.stdout:206158430521 2026-03-31T21:19:30.848 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:30.851 INFO:teuthology.orchestra.run.vm00.stdout:227633266996 2026-03-31T21:19:30.851 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:30.852 INFO:teuthology.orchestra.run.vm00.stdout:90194313575 2026-03-31T21:19:30.852 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:31.056 INFO:teuthology.orchestra.run.vm00.stdout:90194313574 2026-03-31T21:19:31.061 INFO:teuthology.orchestra.run.vm00.stdout:227633266995 2026-03-31T21:19:31.066 INFO:teuthology.orchestra.run.vm00.stdout:206158430520 2026-03-31T21:19:31.077 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266996 got 227633266995 for osd.2 2026-03-31T21:19:31.078 INFO:teuthology.orchestra.run.vm00.stdout:163208757567 2026-03-31T21:19:31.078 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313575 got 90194313574 for osd.1 2026-03-31T21:19:31.087 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430521 got 206158430520 for osd.3 2026-03-31T21:19:31.092 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757568 got 163208757567 for osd.0 2026-03-31T21:19:32.077 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:32.079 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:32.087 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:32.093 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:32.292 INFO:teuthology.orchestra.run.vm00.stdout:227633266997 2026-03-31T21:19:32.308 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266996 got 227633266997 for osd.2 2026-03-31T21:19:32.308 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:32.324 INFO:teuthology.orchestra.run.vm00.stdout:163208757568 2026-03-31T21:19:32.338 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757568 got 163208757568 for osd.0 2026-03-31T21:19:32.338 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:32.342 INFO:teuthology.orchestra.run.vm00.stdout:206158430521 2026-03-31T21:19:32.343 INFO:teuthology.orchestra.run.vm00.stdout:90194313575 2026-03-31T21:19:32.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430521 got 206158430521 for osd.3 2026-03-31T21:19:32.355 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:32.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313575 got 90194313575 for osd.1 2026-03-31T21:19:32.355 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:32.355 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:32.580 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:32.581 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:32.591 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2026-03-31T21:19:31.934864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4620,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22236,"num_bytes_recovered":182430896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":375540,"kb_used_data":143932,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282739980,"statfs":{"total":289910292480,"available":289525739520,"internally_reserved":0,"allocated":147386368,"data_stored":146512514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-105,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":860160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001526"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":58039,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:19:30.747383+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:19:30.747383+0000","last_peered":"2026-03-31T21:19:30.747383+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:30.747383+0000","last_undegraded":"2026-03-31T21:19:30.747383+0000","last_fullsized":"2026-03-31T21:19:30.747383+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4620,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22234,"num_bytes_recovered":182102704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":56,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:15:37.583355+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:15:37.583355+0000","last_peered":"2026-03-31T21:15:37.583355+0000","last_clean":"2026-03-31T21:15:37.583355+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:15:37.583355+0000","last_undegraded":"2026-03-31T21:15:37.583355+0000","last_fullsized":"2026-03-31T21:15:37.583355+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4620,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22234,"num_bytes_recovered":182102704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200077312,"data_stored":200077312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430521,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266997,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757568,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101372,"kb_used_data":35756,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94270468,"statfs":{"total":96636764160,"available":96532959232,"internally_reserved":0,"allocated":36614144,"data_stored":36320470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":35831808,"data_stored":35831808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:32.591 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 58039, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:19:30.747383+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:19:30.747383+0000', 'last_peered': '2026-03-31T21:19:30.747383+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:30.747383+0000', 'last_undegraded': '2026-03-31T21:19:30.747383+0000', 'last_fullsized': '2026-03-31T21:19:30.747383+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4620, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22234, 'num_bytes_recovered': 182102704, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:32.591 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:37.592 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:37.593 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:37.593 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:37.593 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:37.698 INFO:teuthology.orchestra.run.vm00.stdout:206158430523 2026-03-31T21:19:37.699 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:37.703 INFO:teuthology.orchestra.run.vm00.stdout:227633266999 2026-03-31T21:19:37.703 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:37.703 INFO:teuthology.orchestra.run.vm00.stdout:90194313577 2026-03-31T21:19:37.703 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:37.707 INFO:teuthology.orchestra.run.vm00.stdout:163208757570 2026-03-31T21:19:37.707 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:37.904 INFO:teuthology.orchestra.run.vm00.stdout:206158430522 2026-03-31T21:19:37.922 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430523 got 206158430522 for osd.3 2026-03-31T21:19:37.956 INFO:teuthology.orchestra.run.vm00.stdout:90194313576 2026-03-31T21:19:37.957 INFO:teuthology.orchestra.run.vm00.stdout:227633266997 2026-03-31T21:19:37.969 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313577 got 90194313576 for osd.1 2026-03-31T21:19:37.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266999 got 227633266997 for osd.2 2026-03-31T21:19:38.005 INFO:teuthology.orchestra.run.vm00.stdout:163208757569 2026-03-31T21:19:38.016 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757570 got 163208757569 for osd.0 2026-03-31T21:19:38.922 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:38.970 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:38.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:39.016 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:39.125 INFO:teuthology.orchestra.run.vm00.stdout:206158430523 2026-03-31T21:19:39.137 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430523 got 206158430523 for osd.3 2026-03-31T21:19:39.137 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:39.183 INFO:teuthology.orchestra.run.vm00.stdout:90194313578 2026-03-31T21:19:39.196 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313577 got 90194313578 for osd.1 2026-03-31T21:19:39.196 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:39.216 INFO:teuthology.orchestra.run.vm00.stdout:227633266999 2026-03-31T21:19:39.226 INFO:teuthology.orchestra.run.vm00.stdout:163208757570 2026-03-31T21:19:39.228 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633266999 got 227633266999 for osd.2 2026-03-31T21:19:39.228 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:39.237 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757570 got 163208757570 for osd.0 2026-03-31T21:19:39.237 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:39.237 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:39.467 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:39.467 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:39.477 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2026-03-31T21:19:37.935620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4553,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22303,"num_bytes_recovered":182979760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":375816,"kb_used_data":144208,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282739704,"statfs":{"total":289910292480,"available":289525456896,"internally_reserved":0,"allocated":147668992,"data_stored":146795138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-124,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":58173,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:19:37.615862+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:19:37.615862+0000","last_peered":"2026-03-31T21:19:37.615862+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:37.615862+0000","last_undegraded":"2026-03-31T21:19:37.615862+0000","last_fullsized":"2026-03-31T21:19:37.615862+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4553,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22301,"num_bytes_recovered":182651568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4553,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22301,"num_bytes_recovered":182651568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200351744,"data_stored":200351744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430523,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633266999,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757570,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101648,"kb_used_data":36032,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94270192,"statfs":{"total":96636764160,"available":96532676608,"internally_reserved":0,"allocated":36896768,"data_stored":36603094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36106240,"data_stored":36106240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:39.477 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 58173, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:19:37.615862+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:19:37.615862+0000', 'last_peered': '2026-03-31T21:19:37.615862+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:37.615862+0000', 'last_undegraded': '2026-03-31T21:19:37.615862+0000', 'last_fullsized': '2026-03-31T21:19:37.615862+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4553, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22301, 'num_bytes_recovered': 182651568, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:39.477 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:44.478 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:44.478 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:44.478 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:44.479 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:44.578 INFO:teuthology.orchestra.run.vm00.stdout:227633267001 2026-03-31T21:19:44.578 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:44.579 INFO:teuthology.orchestra.run.vm00.stdout:206158430525 2026-03-31T21:19:44.579 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:44.580 INFO:teuthology.orchestra.run.vm00.stdout:90194313580 2026-03-31T21:19:44.580 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:44.589 INFO:teuthology.orchestra.run.vm00.stdout:163208757572 2026-03-31T21:19:44.589 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:44.788 INFO:teuthology.orchestra.run.vm00.stdout:227633267000 2026-03-31T21:19:44.791 INFO:teuthology.orchestra.run.vm00.stdout:206158430524 2026-03-31T21:19:44.805 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267001 got 227633267000 for osd.2 2026-03-31T21:19:44.805 INFO:teuthology.orchestra.run.vm00.stdout:163208757571 2026-03-31T21:19:44.806 INFO:teuthology.orchestra.run.vm00.stdout:90194313579 2026-03-31T21:19:44.809 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430525 got 206158430524 for osd.3 2026-03-31T21:19:44.818 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313580 got 90194313579 for osd.1 2026-03-31T21:19:44.821 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757572 got 163208757571 for osd.0 2026-03-31T21:19:45.805 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:45.810 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:45.818 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:45.822 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:46.018 INFO:teuthology.orchestra.run.vm00.stdout:227633267000 2026-03-31T21:19:46.040 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267001 got 227633267000 for osd.2 2026-03-31T21:19:46.066 INFO:teuthology.orchestra.run.vm00.stdout:90194313579 2026-03-31T21:19:46.069 INFO:teuthology.orchestra.run.vm00.stdout:206158430524 2026-03-31T21:19:46.072 INFO:teuthology.orchestra.run.vm00.stdout:163208757571 2026-03-31T21:19:46.080 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313580 got 90194313579 for osd.1 2026-03-31T21:19:46.084 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430525 got 206158430524 for osd.3 2026-03-31T21:19:46.085 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757572 got 163208757571 for osd.0 2026-03-31T21:19:47.041 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:47.080 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:47.084 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:47.085 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:47.240 INFO:teuthology.orchestra.run.vm00.stdout:227633267001 2026-03-31T21:19:47.256 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267001 got 227633267001 for osd.2 2026-03-31T21:19:47.256 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:47.287 INFO:teuthology.orchestra.run.vm00.stdout:206158430526 2026-03-31T21:19:47.293 INFO:teuthology.orchestra.run.vm00.stdout:90194313580 2026-03-31T21:19:47.299 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430525 got 206158430526 for osd.3 2026-03-31T21:19:47.299 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:47.307 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313580 got 90194313580 for osd.1 2026-03-31T21:19:47.307 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:47.342 INFO:teuthology.orchestra.run.vm00.stdout:163208757573 2026-03-31T21:19:47.352 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757572 got 163208757573 for osd.0 2026-03-31T21:19:47.352 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:47.353 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:47.573 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:47.573 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:47.583 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2026-03-31T21:19:45.936595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4484,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22372,"num_bytes_recovered":183545008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376084,"kb_used_data":144476,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282739436,"statfs":{"total":289910292480,"available":289525182464,"internally_reserved":0,"allocated":147943424,"data_stored":147069570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-136,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001475"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":58311,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:19:44.696896+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:19:44.696896+0000","last_peered":"2026-03-31T21:19:44.696896+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:44.696896+0000","last_undegraded":"2026-03-31T21:19:44.696896+0000","last_fullsized":"2026-03-31T21:19:44.696896+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4484,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22370,"num_bytes_recovered":183216816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4484,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22370,"num_bytes_recovered":183216816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200634368,"data_stored":200634368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430526,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267001,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313580,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757573,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101916,"kb_used_data":36300,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269924,"statfs":{"total":96636764160,"available":96532402176,"internally_reserved":0,"allocated":37171200,"data_stored":36877526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36388864,"data_stored":36388864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:47.583 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 58311, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:19:44.696896+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:19:44.696896+0000', 'last_peered': '2026-03-31T21:19:44.696896+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:44.696896+0000', 'last_undegraded': '2026-03-31T21:19:44.696896+0000', 'last_fullsized': '2026-03-31T21:19:44.696896+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4484, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22370, 'num_bytes_recovered': 183216816, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:47.583 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:52.584 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:52.584 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:52.584 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:52.584 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:52.687 INFO:teuthology.orchestra.run.vm00.stdout:227633267004 2026-03-31T21:19:52.687 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:52.693 INFO:teuthology.orchestra.run.vm00.stdout:163208757575 2026-03-31T21:19:52.693 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:52.695 INFO:teuthology.orchestra.run.vm00.stdout:90194313582 2026-03-31T21:19:52.695 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:52.887 INFO:teuthology.orchestra.run.vm00.stdout:227633267003 2026-03-31T21:19:52.892 INFO:teuthology.orchestra.run.vm00.stdout:206158430528 2026-03-31T21:19:52.892 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:52.904 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267004 got 227633267003 for osd.2 2026-03-31T21:19:52.928 INFO:teuthology.orchestra.run.vm00.stdout:163208757574 2026-03-31T21:19:52.940 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757575 got 163208757574 for osd.0 2026-03-31T21:19:52.943 INFO:teuthology.orchestra.run.vm00.stdout:90194313581 2026-03-31T21:19:52.956 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313582 got 90194313581 for osd.1 2026-03-31T21:19:53.092 INFO:teuthology.orchestra.run.vm00.stdout:206158430527 2026-03-31T21:19:53.104 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430528 got 206158430527 for osd.3 2026-03-31T21:19:53.905 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:53.941 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:53.956 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:54.105 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:54.105 INFO:teuthology.orchestra.run.vm00.stdout:227633267004 2026-03-31T21:19:54.122 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267004 got 227633267004 for osd.2 2026-03-31T21:19:54.122 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:54.159 INFO:teuthology.orchestra.run.vm00.stdout:163208757575 2026-03-31T21:19:54.172 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757575 got 163208757575 for osd.0 2026-03-31T21:19:54.172 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:54.191 INFO:teuthology.orchestra.run.vm00.stdout:90194313583 2026-03-31T21:19:54.206 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313582 got 90194313583 for osd.1 2026-03-31T21:19:54.206 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:54.300 INFO:teuthology.orchestra.run.vm00.stdout:206158430528 2026-03-31T21:19:54.310 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430528 got 206158430528 for osd.3 2026-03-31T21:19:54.310 DEBUG:teuthology.parallel:result is None 2026-03-31T21:19:54.310 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:19:54.495 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:19:54.496 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:19:54.505 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2026-03-31T21:19:53.937642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4405,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22451,"num_bytes_recovered":184192176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376396,"kb_used_data":144788,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282739124,"statfs":{"total":289910292480,"available":289524862976,"internally_reserved":0,"allocated":148262912,"data_stored":147389058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-127,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":1040384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":58469,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:19:52.779955+0000","last_change":"2026-03-31T21:04:14.042275+0000","last_active":"2026-03-31T21:19:52.779955+0000","last_peered":"2026-03-31T21:19:52.779955+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:52.779955+0000","last_undegraded":"2026-03-31T21:19:52.779955+0000","last_fullsized":"2026-03-31T21:19:52.779955+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4405,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22449,"num_bytes_recovered":183863984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4405,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22449,"num_bytes_recovered":183863984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":200953856,"data_stored":200953856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430528,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757575,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102228,"kb_used_data":36612,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269612,"statfs":{"total":96636764160,"available":96532082688,"internally_reserved":0,"allocated":37490688,"data_stored":37197014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36708352,"data_stored":36708352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:19:54.505 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 58469, 'reported_epoch': 57, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-03-31T21:19:52.779955+0000', 'last_change': '2026-03-31T21:04:14.042275+0000', 'last_active': '2026-03-31T21:19:52.779955+0000', 'last_peered': '2026-03-31T21:19:52.779955+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:52.779955+0000', 'last_undegraded': '2026-03-31T21:19:52.779955+0000', 'last_fullsized': '2026-03-31T21:19:52.779955+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4405, 'num_objects_unfound': 0, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22449, 'num_bytes_recovered': 183863984, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 13367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:19:54.505 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-03-31T21:19:58.921 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:19:58.919+0000 7f1d3988f640 -1 osd.3 pg_epoch: 57 pg[2.0s0( v 22'13367 (22'13341,22'13367] local-lis/les=56/57 n=13367 ec=15/15 lis/c=56/53 les/c/f=57/54/0 sis=56) [3,1,0]/[3,1,2]p3(0) backfill=[0(2)] r=0 lpr=56 pi=[53,56)/1 crt=22'13367 lcod 0'0 mlcod 0'0 active+remapped+backfilling rops=1 mbc={0={},1={},2={}}] ECBackendL continue_recovery_op: 2:ad551702:::test:head has inconsistent hinfo 2026-03-31T21:19:58.922 INFO:tasks.ceph.osd.3.vm00.stderr:2026-03-31T21:19:58.919+0000 7f1d3988f640 -1 log_channel(cluster) log [ERR] : 2.0s0 missing primary copy of 2:ad551702:::test:head, will try copies on 1(1),2(2) 2026-03-31T21:19:59.023 INFO:tasks.ceph.osd.2.vm00.stderr:2026-03-31T21:19:59.020+0000 7f272785a640 -1 log_channel(cluster) log [ERR] : Corruption detected: object 2:ad551702:::test:head is missing hash_info 2026-03-31T21:19:59.506 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:19:59.506 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:19:59.506 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:19:59.507 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:19:59.608 INFO:teuthology.orchestra.run.vm00.stdout:90194313585 2026-03-31T21:19:59.608 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:19:59.611 INFO:teuthology.orchestra.run.vm00.stdout:206158430530 2026-03-31T21:19:59.611 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:19:59.614 INFO:teuthology.orchestra.run.vm00.stdout:163208757577 2026-03-31T21:19:59.614 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:19:59.811 INFO:teuthology.orchestra.run.vm00.stdout:227633267006 2026-03-31T21:19:59.811 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:19:59.818 INFO:teuthology.orchestra.run.vm00.stdout:90194313584 2026-03-31T21:19:59.836 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313585 got 90194313584 for osd.1 2026-03-31T21:19:59.852 INFO:teuthology.orchestra.run.vm00.stdout:206158430529 2026-03-31T21:19:59.860 INFO:teuthology.orchestra.run.vm00.stdout:163208757576 2026-03-31T21:19:59.868 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430530 got 206158430529 for osd.3 2026-03-31T21:19:59.879 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757577 got 163208757576 for osd.0 2026-03-31T21:20:00.006 INFO:teuthology.orchestra.run.vm00.stdout:227633267005 2026-03-31T21:20:00.017 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267006 got 227633267005 for osd.2 2026-03-31T21:20:00.020 INFO:tasks.ceph.mon.a.vm00.stderr:2026-03-31T21:20:00.018+0000 7fd709411640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg backfill_unfound (PG_DAMAGED) 2026-03-31T21:20:00.837 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:20:00.870 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:20:00.879 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:20:01.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:20:01.039 INFO:teuthology.orchestra.run.vm00.stdout:90194313585 2026-03-31T21:20:01.055 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313585 got 90194313585 for osd.1 2026-03-31T21:20:01.055 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:01.078 INFO:teuthology.orchestra.run.vm00.stdout:163208757578 2026-03-31T21:20:01.087 INFO:teuthology.orchestra.run.vm00.stdout:206158430531 2026-03-31T21:20:01.089 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757577 got 163208757578 for osd.0 2026-03-31T21:20:01.089 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:01.098 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430530 got 206158430531 for osd.3 2026-03-31T21:20:01.098 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:01.299 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:20:01.297+0000 7fe01494c640 0 --2- 192.168.123.100:0/1037994206 >> v2:192.168.123.100:3300/0 conn(0x7fe01007eb70 0x7fe01007ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:20:01.419 INFO:teuthology.orchestra.run.vm00.stdout:227633267006 2026-03-31T21:20:01.429 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267006 got 227633267006 for osd.2 2026-03-31T21:20:01.429 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:01.429 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:01.607 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:01.607 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:01.617 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":623,"stamp":"2026-03-31T21:19:59.938337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22510,"num_bytes_recovered":184675504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376660,"kb_used_data":145052,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738860,"statfs":{"total":289910292480,"available":289524592640,"internally_reserved":0,"allocated":148533248,"data_stored":147659394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-141,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":58589,"reported_epoch":57,"state":"active+backfill_unfound+degraded+remapped","last_fresh":"2026-03-31T21:19:59.122563+0000","last_change":"2026-03-31T21:19:59.792710+0000","last_active":"2026-03-31T21:19:59.122563+0000","last_peered":"2026-03-31T21:19:59.122563+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:59.122563+0000","last_undegraded":"2026-03-31T21:19:59.122563+0000","last_fullsized":"2026-03-31T21:19:59.122563+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22508,"num_bytes_recovered":184347312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":13366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22508,"num_bytes_recovered":184347312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201199616,"data_stored":201199616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757578,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102492,"kb_used_data":36876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269348,"statfs":{"total":96636764160,"available":96531812352,"internally_reserved":0,"allocated":37761024,"data_stored":37467350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36954112,"data_stored":36954112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:01.617 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 58589, 'reported_epoch': 57, 'state': 'active+backfill_unfound+degraded+remapped', 'last_fresh': '2026-03-31T21:19:59.122563+0000', 'last_change': '2026-03-31T21:19:59.792710+0000', 'last_active': '2026-03-31T21:19:59.122563+0000', 'last_peered': '2026-03-31T21:19:59.122563+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:59.122563+0000', 'last_undegraded': '2026-03-31T21:19:59.122563+0000', 'last_fullsized': '2026-03-31T21:19:59.122563+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 4343, 'num_objects_unfound': 1, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22508, 'num_bytes_recovered': 184347312, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['1(1)'], 'object_location_counts': [{'shards': '1(1)', 'objects': 1}, {'shards': '1(1),2(2),3(0)', 'objects': 13366}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:20:01.617 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-03-31T21:20:01.617 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:20:01.617 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:20:01.617 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:20:01.617 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:20:01.717 INFO:teuthology.orchestra.run.vm00.stdout:90194313586 2026-03-31T21:20:01.717 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:20:01.725 INFO:teuthology.orchestra.run.vm00.stdout:206158430532 2026-03-31T21:20:01.725 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:20:01.725 INFO:teuthology.orchestra.run.vm00.stdout:163208757579 2026-03-31T21:20:01.725 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:20:01.730 INFO:teuthology.orchestra.run.vm00.stdout:227633267008 2026-03-31T21:20:01.730 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:20:01.928 INFO:teuthology.orchestra.run.vm00.stdout:90194313585 2026-03-31T21:20:01.928 INFO:teuthology.orchestra.run.vm00.stdout:227633267006 2026-03-31T21:20:01.931 INFO:teuthology.orchestra.run.vm00.stdout:206158430531 2026-03-31T21:20:01.931 INFO:teuthology.orchestra.run.vm00.stdout:163208757578 2026-03-31T21:20:01.944 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267008 got 227633267006 for osd.2 2026-03-31T21:20:01.944 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313586 got 90194313585 for osd.1 2026-03-31T21:20:01.945 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430532 got 206158430531 for osd.3 2026-03-31T21:20:01.946 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757579 got 163208757578 for osd.0 2026-03-31T21:20:02.944 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:20:02.944 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:20:02.946 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:20:02.946 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:20:03.146 INFO:teuthology.orchestra.run.vm00.stdout:227633267008 2026-03-31T21:20:03.149 INFO:teuthology.orchestra.run.vm00.stdout:206158430532 2026-03-31T21:20:03.151 INFO:teuthology.orchestra.run.vm00.stdout:90194313586 2026-03-31T21:20:03.153 INFO:teuthology.orchestra.run.vm00.stdout:163208757579 2026-03-31T21:20:03.158 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267008 got 227633267008 for osd.2 2026-03-31T21:20:03.158 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:03.165 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430532 got 206158430532 for osd.3 2026-03-31T21:20:03.165 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:03.165 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313586 got 90194313586 for osd.1 2026-03-31T21:20:03.165 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:03.170 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757579 got 163208757579 for osd.0 2026-03-31T21:20:03.170 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:03.170 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:03.397 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:03.398 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:03.408 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2026-03-31T21:20:01.938595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22510,"num_bytes_recovered":184675504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376660,"kb_used_data":145052,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738860,"statfs":{"total":289910292480,"available":289524592640,"internally_reserved":0,"allocated":148533248,"data_stored":147659394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-92,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":58589,"reported_epoch":57,"state":"active+backfill_unfound+degraded+remapped","last_fresh":"2026-03-31T21:19:59.122563+0000","last_change":"2026-03-31T21:19:59.792710+0000","last_active":"2026-03-31T21:19:59.122563+0000","last_peered":"2026-03-31T21:19:59.122563+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:59.122563+0000","last_undegraded":"2026-03-31T21:19:59.122563+0000","last_fullsized":"2026-03-31T21:19:59.122563+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22508,"num_bytes_recovered":184347312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":13366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22508,"num_bytes_recovered":184347312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201199616,"data_stored":201199616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430532,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102492,"kb_used_data":36876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269348,"statfs":{"total":96636764160,"available":96531812352,"internally_reserved":0,"allocated":37761024,"data_stored":37467350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36954112,"data_stored":36954112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:03.408 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "22'13367", 'reported_seq': 58589, 'reported_epoch': 57, 'state': 'active+backfill_unfound+degraded+remapped', 'last_fresh': '2026-03-31T21:19:59.122563+0000', 'last_change': '2026-03-31T21:19:59.792710+0000', 'last_active': '2026-03-31T21:19:59.122563+0000', 'last_peered': '2026-03-31T21:19:59.122563+0000', 'last_clean': '2026-03-31T21:04:08.969311+0000', 'last_became_active': '2026-03-31T21:04:14.041366+0000', 'last_became_peered': '2026-03-31T21:04:14.041366+0000', 'last_unstale': '2026-03-31T21:19:59.122563+0000', 'last_undegraded': '2026-03-31T21:19:59.122563+0000', 'last_fullsized': '2026-03-31T21:19:59.122563+0000', 'mapping_epoch': 56, 'log_start': "22'13341", 'ondisk_log_start': "22'13341", 'created': 15, 'last_epoch_clean': 54, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "18'2", 'last_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_deep_scrub': "18'2", 'last_deep_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'last_clean_scrub_stamp': '2026-03-31T21:01:39.870036+0000', 'objects_scrubbed': 2, 'log_size': 26, 'log_dups_size': 3000, 'ondisk_log_size': 26, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 2, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 109479268, 'num_objects': 13367, 'num_object_clones': 0, 'num_object_copies': 40101, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 4343, 'num_objects_unfound': 1, 'num_objects_dirty': 13367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13367, 'num_write_kb': 106915, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 22508, 'num_bytes_recovered': 184347312, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['1(1)'], 'object_location_counts': [{'shards': '1(1)', 'objects': 1}, {'shards': '1(1),2(2),3(0)', 'objects': 13366}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-03-31T21:20:03.408 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-03-31T21:20:03.636 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:03.636 INFO:teuthology.orchestra.run.vm00.stdout:{"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"1/13369 objects unfound (0.007%)","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 1 pg backfill_unfound","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/40105 objects degraded (0.005%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1146,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1774991047,"num_in_osds":3,"osd_in_since":1774991051,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+backfill_unfound+degraded+remapped","count":1}],"num_pgs":2,"num_pools":2,"num_objects":13369,"data_bytes":109807460,"bytes_used":385699840,"bytes_avail":289524592640,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":40105,"degraded_ratio":4.9869093629223291e-05,"misplaced_objects":4343,"misplaced_total":40105,"misplaced_ratio":0.10829073681585837,"unfound_objects":1,"unfound_total":13369,"unfound_ratio":7.4799910240107707e-05,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":60749,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2026-03-31T21:00:56:876132+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":20,"modified":"2026-03-31T21:18:03.923762+0000","services":{}},"progress_events":{}} 2026-03-31T21:20:03.647 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'edf0be71-aa1c-40ef-819b-fbd038c1604f', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '1/13369 objects unfound (0.007%)', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 1 pg backfill_unfound', 'count': 1}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 2/40105 objects degraded (0.005%), 1 pg degraded', 'count': 1}, 'muted': False}}, 'mutes': []}, 'election_epoch': 4, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 1146, 'monmap': {'epoch': 1, 'min_mon_release_name': 'tentacle', 'num_mons': 3}, 'osdmap': {'epoch': 57, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1774991047, 'num_in_osds': 3, 'osd_in_since': 1774991051, 'num_remapped_pgs': 1}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 1}, {'state_name': 'active+backfill_unfound+degraded+remapped', 'count': 1}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 13369, 'data_bytes': 109807460, 'bytes_used': 385699840, 'bytes_avail': 289524592640, 'bytes_total': 289910292480, 'degraded_objects': 2, 'degraded_total': 40105, 'degraded_ratio': 4.986909362922329e-05, 'misplaced_objects': 4343, 'misplaced_total': 40105, 'misplaced_ratio': 0.10829073681585837, 'unfound_objects': 1, 'unfound_total': 13369, 'unfound_ratio': 7.479991024010771e-05, 'recovering_objects_per_sec': 7, 'recovering_bytes_per_sec': 60749, 'recovering_keys_per_sec': 0, 'num_objects_recovered': 89, 'num_bytes_recovered': 729088, 'num_keys_recovered': 0}, 'fsmap': {'epoch': 1, 'btime': '2026-03-31T21:00:56:876132+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs'], 'services': {}}, 'servicemap': {'epoch': 20, 'modified': '2026-03-31T21:18:03.923762+0000', 'services': {}}, 'progress_events': {}} 2026-03-31T21:20:03.647 DEBUG:tasks.ec_inconsistent_hinfo:there are 1 unfound objects 2026-03-31T21:20:03.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.0 list_unfound '{}' 2026-03-31T21:20:03.726 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:20:03.726 INFO:teuthology.orchestra.run.vm00.stdout: "offset": { 2026-03-31T21:20:03.726 INFO:teuthology.orchestra.run.vm00.stdout: "oid": "", 2026-03-31T21:20:03.726 INFO:teuthology.orchestra.run.vm00.stdout: "key": "", 2026-03-31T21:20:03.726 INFO:teuthology.orchestra.run.vm00.stdout: "snapid": 0, 2026-03-31T21:20:03.726 INFO:teuthology.orchestra.run.vm00.stdout: "hash": 0, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "max": 0, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "pool": -9223372036854775808, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "namespace": "" 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: }, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "num_missing": 1, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "num_unfound": 1, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "objects": [ 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: { 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "oid": { 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "oid": "test", 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "key": "", 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "snapid": -2, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "hash": 1088989877, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "max": 0, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "pool": 2, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "namespace": "" 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: }, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "need": "18'2", 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "have": "0'0", 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "flags": "none", 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "locations": [ 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "1(1)" 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: ] 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: } 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: ], 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "state": "NotBackfilling", 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "available_might_have_unfound": true, 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "might_have_unfound": [], 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout: "more": false 2026-03-31T21:20:03.727 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:20:03.734 DEBUG:tasks.ec_inconsistent_hinfo:list_pg_unfound={'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'test', 'key': '', 'snapid': -2, 'hash': 1088989877, 'max': 0, 'pool': 2, 'namespace': ''}, 'need': "18'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['1(1)']}], 'state': 'NotBackfilling', 'available_might_have_unfound': True, 'might_have_unfound': []} 2026-03-31T21:20:03.734 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:03.915 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:03.915 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:03.925 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2026-03-31T21:20:01.938595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22510,"num_bytes_recovered":184675504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376660,"kb_used_data":145052,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738860,"statfs":{"total":289910292480,"available":289524592640,"internally_reserved":0,"allocated":148533248,"data_stored":147659394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-92,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001472"},"pg_stats":[{"pgid":"2.0","version":"22'13367","reported_seq":58589,"reported_epoch":57,"state":"active+backfill_unfound+degraded+remapped","last_fresh":"2026-03-31T21:19:59.122563+0000","last_change":"2026-03-31T21:19:59.792710+0000","last_active":"2026-03-31T21:19:59.122563+0000","last_peered":"2026-03-31T21:19:59.122563+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:19:59.122563+0000","last_undegraded":"2026-03-31T21:19:59.122563+0000","last_fullsized":"2026-03-31T21:19:59.122563+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":26,"log_dups_size":3000,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22508,"num_bytes_recovered":184347312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":13366}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4343,"num_objects_unfound":1,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22508,"num_bytes_recovered":184347312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201199616,"data_stored":201199616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":26,"ondisk_log_size":26,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430532,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84220,"kb_used_data":53932,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287620,"statfs":{"total":96636764160,"available":96550522880,"internally_reserved":0,"allocated":55226368,"data_stored":54936278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267008,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757579,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102492,"kb_used_data":36876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269348,"statfs":{"total":96636764160,"available":96531812352,"internally_reserved":0,"allocated":37761024,"data_stored":37467350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36954112,"data_stored":36954112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54751232,"data_stored":54751232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:03.926 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.0 mark_unfound_lost delete 2026-03-31T21:20:04.007 INFO:teuthology.orchestra.run.vm00.stderr:pg has 1 objects unfound and apparently lost marking 2026-03-31T21:20:04.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-03-31T21:20:04.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-03-31T21:20:04.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-03-31T21:20:04.015 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-03-31T21:20:04.125 INFO:teuthology.orchestra.run.vm00.stdout:227633267009 2026-03-31T21:20:04.125 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:20:04.130 INFO:teuthology.orchestra.run.vm00.stdout:163208757580 2026-03-31T21:20:04.130 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:20:04.134 INFO:teuthology.orchestra.run.vm00.stdout:90194313588 2026-03-31T21:20:04.134 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:20:04.137 INFO:teuthology.orchestra.run.vm00.stdout:206158430533 2026-03-31T21:20:04.137 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:20:04.340 INFO:teuthology.orchestra.run.vm00.stdout:90194313587 2026-03-31T21:20:04.343 INFO:teuthology.orchestra.run.vm00.stdout:163208757579 2026-03-31T21:20:04.344 INFO:teuthology.orchestra.run.vm00.stdout:227633267008 2026-03-31T21:20:04.355 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313588 got 90194313587 for osd.1 2026-03-31T21:20:04.357 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267009 got 227633267008 for osd.2 2026-03-31T21:20:04.359 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757580 got 163208757579 for osd.0 2026-03-31T21:20:04.391 INFO:teuthology.orchestra.run.vm00.stdout:206158430532 2026-03-31T21:20:04.402 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430533 got 206158430532 for osd.3 2026-03-31T21:20:05.356 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:20:05.358 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:20:05.359 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:20:05.403 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:20:05.553 INFO:teuthology.orchestra.run.vm00.stdout:90194313587 2026-03-31T21:20:05.569 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313588 got 90194313587 for osd.1 2026-03-31T21:20:05.602 INFO:teuthology.orchestra.run.vm00.stdout:227633267008 2026-03-31T21:20:05.617 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267009 got 227633267008 for osd.2 2026-03-31T21:20:05.626 INFO:teuthology.orchestra.run.vm00.stdout:163208757579 2026-03-31T21:20:05.640 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757580 got 163208757579 for osd.0 2026-03-31T21:20:05.657 INFO:teuthology.orchestra.run.vm00.stdout:206158430532 2026-03-31T21:20:05.668 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430533 got 206158430532 for osd.3 2026-03-31T21:20:06.570 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-03-31T21:20:06.618 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-03-31T21:20:06.641 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-03-31T21:20:06.668 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-03-31T21:20:06.771 INFO:teuthology.orchestra.run.vm00.stdout:90194313588 2026-03-31T21:20:06.787 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 90194313588 got 90194313588 for osd.1 2026-03-31T21:20:06.788 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:06.807 INFO:teuthology.orchestra.run.vm00.stdout:227633267009 2026-03-31T21:20:06.820 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 227633267009 got 227633267009 for osd.2 2026-03-31T21:20:06.820 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:06.864 INFO:teuthology.orchestra.run.vm00.stdout:163208757581 2026-03-31T21:20:06.876 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 163208757580 got 163208757581 for osd.0 2026-03-31T21:20:06.876 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:06.898 INFO:teuthology.orchestra.run.vm00.stdout:206158430534 2026-03-31T21:20:06.909 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 206158430533 got 206158430534 for osd.3 2026-03-31T21:20:06.909 DEBUG:teuthology.parallel:result is None 2026-03-31T21:20:06.909 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for recovery to complete 2026-03-31T21:20:06.909 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:07.131 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:07.131 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:07.141 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2026-03-31T21:20:05.939140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22517,"num_bytes_recovered":184724656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376656,"kb_used_data":145048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738864,"statfs":{"total":289910292480,"available":289524596736,"internally_reserved":0,"allocated":148529152,"data_stored":147655298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-66,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58610,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:04.723452+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:04.723452+0000","last_peered":"2026-03-31T21:20:04.723452+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:04.723452+0000","last_undegraded":"2026-03-31T21:20:04.723452+0000","last_fullsized":"2026-03-31T21:20:04.723452+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22515,"num_bytes_recovered":184396464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22515,"num_bytes_recovered":184396464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201216000,"data_stored":201216000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757581,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102492,"kb_used_data":36876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269348,"statfs":{"total":96636764160,"available":96531812352,"internally_reserved":0,"allocated":37761024,"data_stored":37467350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36978688,"data_stored":36978688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:07.141 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:07.322 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:07.322 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:07.332 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2026-03-31T21:20:05.939140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22517,"num_bytes_recovered":184724656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376656,"kb_used_data":145048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738864,"statfs":{"total":289910292480,"available":289524596736,"internally_reserved":0,"allocated":148529152,"data_stored":147655298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-66,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58610,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:04.723452+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:04.723452+0000","last_peered":"2026-03-31T21:20:04.723452+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:04.723452+0000","last_undegraded":"2026-03-31T21:20:04.723452+0000","last_fullsized":"2026-03-31T21:20:04.723452+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22515,"num_bytes_recovered":184396464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22515,"num_bytes_recovered":184396464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201216000,"data_stored":201216000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757581,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102492,"kb_used_data":36876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269348,"statfs":{"total":96636764160,"available":96531812352,"internally_reserved":0,"allocated":37761024,"data_stored":37467350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36978688,"data_stored":36978688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:07.332 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:07.514 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:07.515 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:07.524 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2026-03-31T21:20:05.939140+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22517,"num_bytes_recovered":184724656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376656,"kb_used_data":145048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738864,"statfs":{"total":289910292480,"available":289524596736,"internally_reserved":0,"allocated":148529152,"data_stored":147655298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-66,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":532480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58610,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:04.723452+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:04.723452+0000","last_peered":"2026-03-31T21:20:04.723452+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:04.723452+0000","last_undegraded":"2026-03-31T21:20:04.723452+0000","last_fullsized":"2026-03-31T21:20:04.723452+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22515,"num_bytes_recovered":184396464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4339,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22515,"num_bytes_recovered":184396464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201216000,"data_stored":201216000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430534,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267009,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189948,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181892,"statfs":{"total":96636764160,"available":96442257408,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757581,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102492,"kb_used_data":36876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269348,"statfs":{"total":96636764160,"available":96531812352,"internally_reserved":0,"allocated":37761024,"data_stored":37467350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36978688,"data_stored":36978688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:10.525 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:10.708 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:10.708 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:10.717 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2026-03-31T21:20:09.939623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22566,"num_bytes_recovered":185126064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376840,"kb_used_data":145240,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738680,"statfs":{"total":289910292480,"available":289524408320,"internally_reserved":0,"allocated":148725760,"data_stored":147851906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58708,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:09.732132+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:09.732132+0000","last_peered":"2026-03-31T21:20:09.732132+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:09.732132+0000","last_undegraded":"2026-03-31T21:20:09.732132+0000","last_fullsized":"2026-03-31T21:20:09.732132+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201416704,"data_stored":201416704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102684,"kb_used_data":37068,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269156,"statfs":{"total":96636764160,"available":96531615744,"internally_reserved":0,"allocated":37957632,"data_stored":37663958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37179392,"data_stored":37179392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:10.718 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:10.900 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:10.900 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:10.910 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2026-03-31T21:20:09.939623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22566,"num_bytes_recovered":185126064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376840,"kb_used_data":145240,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738680,"statfs":{"total":289910292480,"available":289524408320,"internally_reserved":0,"allocated":148725760,"data_stored":147851906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58708,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:09.732132+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:09.732132+0000","last_peered":"2026-03-31T21:20:09.732132+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:09.732132+0000","last_undegraded":"2026-03-31T21:20:09.732132+0000","last_fullsized":"2026-03-31T21:20:09.732132+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201416704,"data_stored":201416704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102684,"kb_used_data":37068,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269156,"statfs":{"total":96636764160,"available":96531615744,"internally_reserved":0,"allocated":37957632,"data_stored":37663958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37179392,"data_stored":37179392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:13.912 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:14.110 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:14.110 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:14.121 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2026-03-31T21:20:13.940122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22566,"num_bytes_recovered":185126064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376840,"kb_used_data":145240,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738680,"statfs":{"total":289910292480,"available":289524408320,"internally_reserved":0,"allocated":148725760,"data_stored":147851906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-53,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":450560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001527"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58708,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:09.732132+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:09.732132+0000","last_peered":"2026-03-31T21:20:09.732132+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:09.732132+0000","last_undegraded":"2026-03-31T21:20:09.732132+0000","last_fullsized":"2026-03-31T21:20:09.732132+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201416704,"data_stored":201416704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102684,"kb_used_data":37068,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269156,"statfs":{"total":96636764160,"available":96531615744,"internally_reserved":0,"allocated":37957632,"data_stored":37663958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37179392,"data_stored":37179392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:14.121 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:14.304 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:14.304 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:14.314 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":630,"stamp":"2026-03-31T21:20:13.940122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22566,"num_bytes_recovered":185126064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":376840,"kb_used_data":145240,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738680,"statfs":{"total":289910292480,"available":289524408320,"internally_reserved":0,"allocated":148725760,"data_stored":147851906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-53,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":450560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001527"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58708,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:09.732132+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:09.732132+0000","last_peered":"2026-03-31T21:20:09.732132+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:09.732132+0000","last_undegraded":"2026-03-31T21:20:09.732132+0000","last_fullsized":"2026-03-31T21:20:09.732132+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4290,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22564,"num_bytes_recovered":184797872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201416704,"data_stored":201416704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430535,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757582,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102684,"kb_used_data":37068,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94269156,"statfs":{"total":96636764160,"available":96531615744,"internally_reserved":0,"allocated":37957632,"data_stored":37663958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37179392,"data_stored":37179392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:17.316 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:17.502 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:17.502 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:17.512 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2026-03-31T21:20:15.940373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4241,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22615,"num_bytes_recovered":185527472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377036,"kb_used_data":145436,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738484,"statfs":{"total":289910292480,"available":289524207616,"internally_reserved":0,"allocated":148926464,"data_stored":148052610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-102,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58806,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:14.741641+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:14.741641+0000","last_peered":"2026-03-31T21:20:14.741641+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:14.741641+0000","last_undegraded":"2026-03-31T21:20:14.741641+0000","last_fullsized":"2026-03-31T21:20:14.741641+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4241,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22613,"num_bytes_recovered":185199280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4241,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22613,"num_bytes_recovered":185199280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201617408,"data_stored":201617408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430536,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102880,"kb_used_data":37264,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268960,"statfs":{"total":96636764160,"available":96531415040,"internally_reserved":0,"allocated":38158336,"data_stored":37864662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37380096,"data_stored":37380096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:17.512 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:17.692 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:17.692 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:17.701 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2026-03-31T21:20:15.940373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4241,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22615,"num_bytes_recovered":185527472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377036,"kb_used_data":145436,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738484,"statfs":{"total":289910292480,"available":289524207616,"internally_reserved":0,"allocated":148926464,"data_stored":148052610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-102,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":851968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001494"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58806,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:14.741641+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:14.741641+0000","last_peered":"2026-03-31T21:20:14.741641+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:14.741641+0000","last_undegraded":"2026-03-31T21:20:14.741641+0000","last_fullsized":"2026-03-31T21:20:14.741641+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4241,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22613,"num_bytes_recovered":185199280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4241,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22613,"num_bytes_recovered":185199280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201617408,"data_stored":201617408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430536,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267011,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":102880,"kb_used_data":37264,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268960,"statfs":{"total":96636764160,"available":96531415040,"internally_reserved":0,"allocated":38158336,"data_stored":37864662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37380096,"data_stored":37380096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:20.702 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:20.885 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:20.885 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:20.895 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2026-03-31T21:20:19.940879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22664,"num_bytes_recovered":185928880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377232,"kb_used_data":145632,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738288,"statfs":{"total":289910292480,"available":289524006912,"internally_reserved":0,"allocated":149127168,"data_stored":148253314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58904,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:19.749395+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:19.749395+0000","last_peered":"2026-03-31T21:20:19.749395+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:19.749395+0000","last_undegraded":"2026-03-31T21:20:19.749395+0000","last_fullsized":"2026-03-31T21:20:19.749395+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201818112,"data_stored":201818112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430537,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103076,"kb_used_data":37460,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268764,"statfs":{"total":96636764160,"available":96531214336,"internally_reserved":0,"allocated":38359040,"data_stored":38065366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37580800,"data_stored":37580800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:20.895 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:21.076 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:21.077 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:21.086 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2026-03-31T21:20:19.940879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22664,"num_bytes_recovered":185928880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377232,"kb_used_data":145632,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738288,"statfs":{"total":289910292480,"available":289524006912,"internally_reserved":0,"allocated":149127168,"data_stored":148253314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58904,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:19.749395+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:19.749395+0000","last_peered":"2026-03-31T21:20:19.749395+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:19.749395+0000","last_undegraded":"2026-03-31T21:20:19.749395+0000","last_fullsized":"2026-03-31T21:20:19.749395+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201818112,"data_stored":201818112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430537,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103076,"kb_used_data":37460,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268764,"statfs":{"total":96636764160,"available":96531214336,"internally_reserved":0,"allocated":38359040,"data_stored":38065366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37580800,"data_stored":37580800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:24.087 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:24.276 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:24.277 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:24.287 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2026-03-31T21:20:23.941462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22664,"num_bytes_recovered":185928880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377232,"kb_used_data":145632,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738288,"statfs":{"total":289910292480,"available":289524006912,"internally_reserved":0,"allocated":149127168,"data_stored":148253314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58904,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:19.749395+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:19.749395+0000","last_peered":"2026-03-31T21:20:19.749395+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:19.749395+0000","last_undegraded":"2026-03-31T21:20:19.749395+0000","last_fullsized":"2026-03-31T21:20:19.749395+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201818112,"data_stored":201818112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430537,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103076,"kb_used_data":37460,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268764,"statfs":{"total":96636764160,"available":96531214336,"internally_reserved":0,"allocated":38359040,"data_stored":38065366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37580800,"data_stored":37580800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:24.287 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:24.483 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:24.484 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:24.494 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2026-03-31T21:20:23.941462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22664,"num_bytes_recovered":185928880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377232,"kb_used_data":145632,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738288,"statfs":{"total":289910292480,"available":289524006912,"internally_reserved":0,"allocated":149127168,"data_stored":148253314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001565"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":58904,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:19.749395+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:19.749395+0000","last_peered":"2026-03-31T21:20:19.749395+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:19.749395+0000","last_undegraded":"2026-03-31T21:20:19.749395+0000","last_fullsized":"2026-03-31T21:20:19.749395+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4192,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22662,"num_bytes_recovered":185600688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":201818112,"data_stored":201818112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430537,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103076,"kb_used_data":37460,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268764,"statfs":{"total":96636764160,"available":96531214336,"internally_reserved":0,"allocated":38359040,"data_stored":38065366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37580800,"data_stored":37580800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:27.495 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:27.685 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:27.685 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:27.695 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-03-31T21:20:25.941718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4143,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22713,"num_bytes_recovered":186330288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377432,"kb_used_data":145832,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738088,"statfs":{"total":289910292480,"available":289523802112,"internally_reserved":0,"allocated":149331968,"data_stored":148458114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001596"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59002,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:24.767871+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:24.767871+0000","last_peered":"2026-03-31T21:20:24.767871+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:24.767871+0000","last_undegraded":"2026-03-31T21:20:24.767871+0000","last_fullsized":"2026-03-31T21:20:24.767871+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22711,"num_bytes_recovered":186002096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22711,"num_bytes_recovered":186002096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202018816,"data_stored":202018816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103276,"kb_used_data":37660,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268564,"statfs":{"total":96636764160,"available":96531009536,"internally_reserved":0,"allocated":38563840,"data_stored":38270166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37781504,"data_stored":37781504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:27.696 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:27.880 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:27.881 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:27.890 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":636,"stamp":"2026-03-31T21:20:25.941718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4143,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22713,"num_bytes_recovered":186330288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377432,"kb_used_data":145832,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282738088,"statfs":{"total":289910292480,"available":289523802112,"internally_reserved":0,"allocated":149331968,"data_stored":148458114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001596"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59002,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:24.767871+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:24.767871+0000","last_peered":"2026-03-31T21:20:24.767871+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:24.767871+0000","last_undegraded":"2026-03-31T21:20:24.767871+0000","last_fullsized":"2026-03-31T21:20:24.767871+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22711,"num_bytes_recovered":186002096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22711,"num_bytes_recovered":186002096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202018816,"data_stored":202018816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430538,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103276,"kb_used_data":37660,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268564,"statfs":{"total":96636764160,"available":96531009536,"internally_reserved":0,"allocated":38563840,"data_stored":38270166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37781504,"data_stored":37781504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:30.891 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:31.076 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:31.077 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:31.087 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-03-31T21:20:29.942200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22762,"num_bytes_recovered":186731696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377624,"kb_used_data":146024,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737896,"statfs":{"total":289910292480,"available":289523605504,"internally_reserved":0,"allocated":149528576,"data_stored":148654722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59100,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:29.784972+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:29.784972+0000","last_peered":"2026-03-31T21:20:29.784972+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:29.784972+0000","last_undegraded":"2026-03-31T21:20:29.784972+0000","last_fullsized":"2026-03-31T21:20:29.784972+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202219520,"data_stored":202219520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103468,"kb_used_data":37852,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268372,"statfs":{"total":96636764160,"available":96530812928,"internally_reserved":0,"allocated":38760448,"data_stored":38466774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37982208,"data_stored":37982208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:31.087 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:31.280 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:31.281 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:31.291 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2026-03-31T21:20:29.942200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22762,"num_bytes_recovered":186731696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377624,"kb_used_data":146024,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737896,"statfs":{"total":289910292480,"available":289523605504,"internally_reserved":0,"allocated":149528576,"data_stored":148654722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59100,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:29.784972+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:29.784972+0000","last_peered":"2026-03-31T21:20:29.784972+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:29.784972+0000","last_undegraded":"2026-03-31T21:20:29.784972+0000","last_fullsized":"2026-03-31T21:20:29.784972+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202219520,"data_stored":202219520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103468,"kb_used_data":37852,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268372,"statfs":{"total":96636764160,"available":96530812928,"internally_reserved":0,"allocated":38760448,"data_stored":38466774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37982208,"data_stored":37982208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:34.292 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:34.476 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:34.476 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:34.487 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2026-03-31T21:20:33.942683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22762,"num_bytes_recovered":186731696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377624,"kb_used_data":146024,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737896,"statfs":{"total":289910292480,"available":289523605504,"internally_reserved":0,"allocated":149528576,"data_stored":148654722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001555"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59100,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:29.784972+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:29.784972+0000","last_peered":"2026-03-31T21:20:29.784972+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:29.784972+0000","last_undegraded":"2026-03-31T21:20:29.784972+0000","last_fullsized":"2026-03-31T21:20:29.784972+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202219520,"data_stored":202219520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103468,"kb_used_data":37852,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268372,"statfs":{"total":96636764160,"available":96530812928,"internally_reserved":0,"allocated":38760448,"data_stored":38466774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37982208,"data_stored":37982208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:34.487 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:34.666 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:34.666 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:34.675 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2026-03-31T21:20:33.942683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22762,"num_bytes_recovered":186731696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377624,"kb_used_data":146024,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737896,"statfs":{"total":289910292480,"available":289523605504,"internally_reserved":0,"allocated":149528576,"data_stored":148654722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001555"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59100,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:29.784972+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:29.784972+0000","last_peered":"2026-03-31T21:20:29.784972+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:29.784972+0000","last_undegraded":"2026-03-31T21:20:29.784972+0000","last_fullsized":"2026-03-31T21:20:29.784972+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4094,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22760,"num_bytes_recovered":186403504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202219520,"data_stored":202219520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430539,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757586,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103468,"kb_used_data":37852,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268372,"statfs":{"total":96636764160,"available":96530812928,"internally_reserved":0,"allocated":38760448,"data_stored":38466774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37982208,"data_stored":37982208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:37.676 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:37.858 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:37.858 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:37.869 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":641,"stamp":"2026-03-31T21:20:35.942911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22809,"num_bytes_recovered":187116720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377816,"kb_used_data":146216,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737704,"statfs":{"total":289910292480,"available":289523408896,"internally_reserved":0,"allocated":149725184,"data_stored":148851330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59194,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:34.794115+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:34.794115+0000","last_peered":"2026-03-31T21:20:34.794115+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:34.794115+0000","last_undegraded":"2026-03-31T21:20:34.794115+0000","last_fullsized":"2026-03-31T21:20:34.794115+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":186788528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":186788528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202412032,"data_stored":202412032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267015,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103660,"kb_used_data":38044,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268180,"statfs":{"total":96636764160,"available":96530616320,"internally_reserved":0,"allocated":38957056,"data_stored":38663382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38174720,"data_stored":38174720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:37.869 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:38.054 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:38.055 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:38.066 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2026-03-31T21:20:37.943122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22809,"num_bytes_recovered":187116720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":377816,"kb_used_data":146216,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737704,"statfs":{"total":289910292480,"available":289523408896,"internally_reserved":0,"allocated":149725184,"data_stored":148851330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001403"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59194,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:34.794115+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:34.794115+0000","last_peered":"2026-03-31T21:20:34.794115+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:34.794115+0000","last_undegraded":"2026-03-31T21:20:34.794115+0000","last_fullsized":"2026-03-31T21:20:34.794115+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":186788528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22807,"num_bytes_recovered":186788528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202412032,"data_stored":202412032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430540,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757587,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103660,"kb_used_data":38044,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94268180,"statfs":{"total":96636764160,"available":96530616320,"internally_reserved":0,"allocated":38957056,"data_stored":38663382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38174720,"data_stored":38174720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:41.067 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:41.251 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:41.251 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:41.262 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2026-03-31T21:20:39.943321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22857,"num_bytes_recovered":187509936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378016,"kb_used_data":146416,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737504,"statfs":{"total":289910292480,"available":289523204096,"internally_reserved":0,"allocated":149929984,"data_stored":149056130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001352"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59290,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:39.703541+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:39.703541+0000","last_peered":"2026-03-31T21:20:39.703541+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:39.703541+0000","last_undegraded":"2026-03-31T21:20:39.703541+0000","last_fullsized":"2026-03-31T21:20:39.703541+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202608640,"data_stored":202608640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103860,"kb_used_data":38244,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267980,"statfs":{"total":96636764160,"available":96530411520,"internally_reserved":0,"allocated":39161856,"data_stored":38868182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38371328,"data_stored":38371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:41.262 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:41.446 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:41.446 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:41.456 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2026-03-31T21:20:39.943321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22857,"num_bytes_recovered":187509936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378016,"kb_used_data":146416,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737504,"statfs":{"total":289910292480,"available":289523204096,"internally_reserved":0,"allocated":149929984,"data_stored":149056130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001352"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59290,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:39.703541+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:39.703541+0000","last_peered":"2026-03-31T21:20:39.703541+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:39.703541+0000","last_undegraded":"2026-03-31T21:20:39.703541+0000","last_fullsized":"2026-03-31T21:20:39.703541+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202608640,"data_stored":202608640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267016,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103860,"kb_used_data":38244,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267980,"statfs":{"total":96636764160,"available":96530411520,"internally_reserved":0,"allocated":39161856,"data_stored":38868182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38371328,"data_stored":38371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:44.456 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:44.642 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:44.642 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:44.652 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2026-03-31T21:20:43.943799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22857,"num_bytes_recovered":187509936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378016,"kb_used_data":146416,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737504,"statfs":{"total":289910292480,"available":289523204096,"internally_reserved":0,"allocated":149929984,"data_stored":149056130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-95,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001366"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59290,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:39.703541+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:39.703541+0000","last_peered":"2026-03-31T21:20:39.703541+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:39.703541+0000","last_undegraded":"2026-03-31T21:20:39.703541+0000","last_fullsized":"2026-03-31T21:20:39.703541+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202608640,"data_stored":202608640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103860,"kb_used_data":38244,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267980,"statfs":{"total":96636764160,"available":96530411520,"internally_reserved":0,"allocated":39161856,"data_stored":38868182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38371328,"data_stored":38371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:44.652 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:44.839 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:44.839 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:44.850 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2026-03-31T21:20:43.943799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22857,"num_bytes_recovered":187509936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378016,"kb_used_data":146416,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737504,"statfs":{"total":289910292480,"available":289523204096,"internally_reserved":0,"allocated":149929984,"data_stored":149056130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-95,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001366"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59290,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:39.703541+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:39.703541+0000","last_peered":"2026-03-31T21:20:39.703541+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:39.703541+0000","last_undegraded":"2026-03-31T21:20:39.703541+0000","last_fullsized":"2026-03-31T21:20:39.703541+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3999,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22855,"num_bytes_recovered":187181744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202608640,"data_stored":202608640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430541,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267017,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103860,"kb_used_data":38244,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267980,"statfs":{"total":96636764160,"available":96530411520,"internally_reserved":0,"allocated":39161856,"data_stored":38868182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38371328,"data_stored":38371328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:47.851 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:48.038 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:48.038 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:48.048 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2026-03-31T21:20:47.944289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3950,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22906,"num_bytes_recovered":187911344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378204,"kb_used_data":146604,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737316,"statfs":{"total":289910292480,"available":289523011584,"internally_reserved":0,"allocated":150122496,"data_stored":149248642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001377"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59388,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:44.716678+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:44.716678+0000","last_peered":"2026-03-31T21:20:44.716678+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:44.716678+0000","last_undegraded":"2026-03-31T21:20:44.716678+0000","last_fullsized":"2026-03-31T21:20:44.716678+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3950,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22904,"num_bytes_recovered":187583152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3950,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22904,"num_bytes_recovered":187583152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202809344,"data_stored":202809344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104048,"kb_used_data":38432,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267792,"statfs":{"total":96636764160,"available":96530219008,"internally_reserved":0,"allocated":39354368,"data_stored":39060694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38572032,"data_stored":38572032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:48.049 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:48.231 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:48.232 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:48.242 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":647,"stamp":"2026-03-31T21:20:47.944289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3950,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22906,"num_bytes_recovered":187911344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378204,"kb_used_data":146604,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737316,"statfs":{"total":289910292480,"available":289523011584,"internally_reserved":0,"allocated":150122496,"data_stored":149248642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001377"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59388,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:44.716678+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:44.716678+0000","last_peered":"2026-03-31T21:20:44.716678+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:44.716678+0000","last_undegraded":"2026-03-31T21:20:44.716678+0000","last_fullsized":"2026-03-31T21:20:44.716678+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3950,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22904,"num_bytes_recovered":187583152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3950,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22904,"num_bytes_recovered":187583152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202809344,"data_stored":202809344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430542,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104048,"kb_used_data":38432,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267792,"statfs":{"total":96636764160,"available":96530219008,"internally_reserved":0,"allocated":39354368,"data_stored":39060694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38572032,"data_stored":38572032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:51.243 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:51.432 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:51.432 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:51.443 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2026-03-31T21:20:49.944505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22955,"num_bytes_recovered":188312752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378404,"kb_used_data":146804,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737116,"statfs":{"total":289910292480,"available":289522806784,"internally_reserved":0,"allocated":150327296,"data_stored":149453442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001383"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59486,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:49.740117+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:49.740117+0000","last_peered":"2026-03-31T21:20:49.740117+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:49.740117+0000","last_undegraded":"2026-03-31T21:20:49.740117+0000","last_fullsized":"2026-03-31T21:20:49.740117+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203010048,"data_stored":203010048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104248,"kb_used_data":38632,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267592,"statfs":{"total":96636764160,"available":96530014208,"internally_reserved":0,"allocated":39559168,"data_stored":39265494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38772736,"data_stored":38772736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:51.443 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:51.631 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:51.631 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:51.642 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2026-03-31T21:20:49.944505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22955,"num_bytes_recovered":188312752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378404,"kb_used_data":146804,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737116,"statfs":{"total":289910292480,"available":289522806784,"internally_reserved":0,"allocated":150327296,"data_stored":149453442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001383"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59486,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:49.740117+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:49.740117+0000","last_peered":"2026-03-31T21:20:49.740117+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:49.740117+0000","last_undegraded":"2026-03-31T21:20:49.740117+0000","last_fullsized":"2026-03-31T21:20:49.740117+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203010048,"data_stored":203010048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313597,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104248,"kb_used_data":38632,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267592,"statfs":{"total":96636764160,"available":96530014208,"internally_reserved":0,"allocated":39559168,"data_stored":39265494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38772736,"data_stored":38772736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:54.643 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:54.827 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:54.828 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:54.837 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":650,"stamp":"2026-03-31T21:20:53.944937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22955,"num_bytes_recovered":188312752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378404,"kb_used_data":146804,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737116,"statfs":{"total":289910292480,"available":289522806784,"internally_reserved":0,"allocated":150327296,"data_stored":149453442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001385"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59486,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:49.740117+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:49.740117+0000","last_peered":"2026-03-31T21:20:49.740117+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:49.740117+0000","last_undegraded":"2026-03-31T21:20:49.740117+0000","last_fullsized":"2026-03-31T21:20:49.740117+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203010048,"data_stored":203010048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104248,"kb_used_data":38632,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267592,"statfs":{"total":96636764160,"available":96530014208,"internally_reserved":0,"allocated":39559168,"data_stored":39265494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38772736,"data_stored":38772736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:54.837 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:55.019 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:55.019 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:55.029 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":650,"stamp":"2026-03-31T21:20:53.944937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22955,"num_bytes_recovered":188312752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378404,"kb_used_data":146804,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282737116,"statfs":{"total":289910292480,"available":289522806784,"internally_reserved":0,"allocated":150327296,"data_stored":149453442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001385"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59486,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:49.740117+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:49.740117+0000","last_peered":"2026-03-31T21:20:49.740117+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:49.740117+0000","last_undegraded":"2026-03-31T21:20:49.740117+0000","last_fullsized":"2026-03-31T21:20:49.740117+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3901,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22953,"num_bytes_recovered":187984560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203010048,"data_stored":203010048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430543,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267019,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104248,"kb_used_data":38632,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267592,"statfs":{"total":96636764160,"available":96530014208,"internally_reserved":0,"allocated":39559168,"data_stored":39265494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38772736,"data_stored":38772736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:58.030 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:58.216 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:58.217 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:58.228 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2026-03-31T21:20:57.945429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23004,"num_bytes_recovered":188714160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378600,"kb_used_data":147000,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736920,"statfs":{"total":289910292480,"available":289522606080,"internally_reserved":0,"allocated":150528000,"data_stored":149654146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001379"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59584,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:54.750928+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:54.750928+0000","last_peered":"2026-03-31T21:20:54.750928+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:54.750928+0000","last_undegraded":"2026-03-31T21:20:54.750928+0000","last_fullsized":"2026-03-31T21:20:54.750928+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23002,"num_bytes_recovered":188385968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23002,"num_bytes_recovered":188385968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203210752,"data_stored":203210752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104444,"kb_used_data":38828,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267396,"statfs":{"total":96636764160,"available":96529813504,"internally_reserved":0,"allocated":39759872,"data_stored":39466198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38973440,"data_stored":38973440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:20:58.228 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:20:58.412 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:20:58.412 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:20:58.423 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2026-03-31T21:20:57.945429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23004,"num_bytes_recovered":188714160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378600,"kb_used_data":147000,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736920,"statfs":{"total":289910292480,"available":289522606080,"internally_reserved":0,"allocated":150528000,"data_stored":149654146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001379"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59584,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:54.750928+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:54.750928+0000","last_peered":"2026-03-31T21:20:54.750928+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:54.750928+0000","last_undegraded":"2026-03-31T21:20:54.750928+0000","last_fullsized":"2026-03-31T21:20:54.750928+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23002,"num_bytes_recovered":188385968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3852,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23002,"num_bytes_recovered":188385968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203210752,"data_stored":203210752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430544,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757591,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104444,"kb_used_data":38828,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267396,"statfs":{"total":96636764160,"available":96529813504,"internally_reserved":0,"allocated":39759872,"data_stored":39466198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38973440,"data_stored":38973440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:01.424 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:01.608 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:01.608 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:01.619 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2026-03-31T21:20:59.945703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23053,"num_bytes_recovered":189115568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378792,"kb_used_data":147192,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736728,"statfs":{"total":289910292480,"available":289522409472,"internally_reserved":0,"allocated":150724608,"data_stored":149850754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59682,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:59.761904+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:59.761904+0000","last_peered":"2026-03-31T21:20:59.761904+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:59.761904+0000","last_undegraded":"2026-03-31T21:20:59.761904+0000","last_fullsized":"2026-03-31T21:20:59.761904+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203411456,"data_stored":203411456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104636,"kb_used_data":39020,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267204,"statfs":{"total":96636764160,"available":96529616896,"internally_reserved":0,"allocated":39956480,"data_stored":39662806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39174144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:01.619 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:01.802 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:01.802 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:01.812 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2026-03-31T21:20:59.945703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23053,"num_bytes_recovered":189115568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378792,"kb_used_data":147192,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736728,"statfs":{"total":289910292480,"available":289522409472,"internally_reserved":0,"allocated":150724608,"data_stored":149850754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001413"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59682,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:59.761904+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:59.761904+0000","last_peered":"2026-03-31T21:20:59.761904+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:59.761904+0000","last_undegraded":"2026-03-31T21:20:59.761904+0000","last_fullsized":"2026-03-31T21:20:59.761904+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203411456,"data_stored":203411456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267020,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104636,"kb_used_data":39020,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267204,"statfs":{"total":96636764160,"available":96529616896,"internally_reserved":0,"allocated":39956480,"data_stored":39662806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39174144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:04.814 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:05.086 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:05.086 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:05.098 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2026-03-31T21:21:03.946165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23053,"num_bytes_recovered":189115568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378792,"kb_used_data":147192,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736728,"statfs":{"total":289910292480,"available":289522409472,"internally_reserved":0,"allocated":150724608,"data_stored":149850754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59682,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:59.761904+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:59.761904+0000","last_peered":"2026-03-31T21:20:59.761904+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:59.761904+0000","last_undegraded":"2026-03-31T21:20:59.761904+0000","last_fullsized":"2026-03-31T21:20:59.761904+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203411456,"data_stored":203411456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104636,"kb_used_data":39020,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267204,"statfs":{"total":96636764160,"available":96529616896,"internally_reserved":0,"allocated":39956480,"data_stored":39662806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39174144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:05.098 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:05.291 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:05.291 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:05.302 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2026-03-31T21:21:03.946165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23053,"num_bytes_recovered":189115568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378792,"kb_used_data":147192,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736728,"statfs":{"total":289910292480,"available":289522409472,"internally_reserved":0,"allocated":150724608,"data_stored":149850754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59682,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:20:59.761904+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:20:59.761904+0000","last_peered":"2026-03-31T21:20:59.761904+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:20:59.761904+0000","last_undegraded":"2026-03-31T21:20:59.761904+0000","last_fullsized":"2026-03-31T21:20:59.761904+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3803,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23051,"num_bytes_recovered":188787376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203411456,"data_stored":203411456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267021,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757592,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104636,"kb_used_data":39020,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267204,"statfs":{"total":96636764160,"available":96529616896,"internally_reserved":0,"allocated":39956480,"data_stored":39662806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39174144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:08.303 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:08.486 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:08.486 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:08.497 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2026-03-31T21:21:07.946648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23102,"num_bytes_recovered":189508808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378988,"kb_used_data":147388,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736532,"statfs":{"total":289910292480,"available":289522208768,"internally_reserved":0,"allocated":150925312,"data_stored":150051458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":794648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59780,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:04.769616+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:04.769616+0000","last_peered":"2026-03-31T21:21:04.769616+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:04.769616+0000","last_undegraded":"2026-03-31T21:21:04.769616+0000","last_fullsized":"2026-03-31T21:21:04.769616+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23100,"num_bytes_recovered":189180616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23100,"num_bytes_recovered":189180616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203612160,"data_stored":203612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104832,"kb_used_data":39216,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267008,"statfs":{"total":96636764160,"available":96529416192,"internally_reserved":0,"allocated":40157184,"data_stored":39863510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39374848,"data_stored":39374848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:08.497 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:08.677 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:08.678 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:08.687 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2026-03-31T21:21:07.946648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23102,"num_bytes_recovered":189508808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":378988,"kb_used_data":147388,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736532,"statfs":{"total":289910292480,"available":289522208768,"internally_reserved":0,"allocated":150925312,"data_stored":150051458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":794648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59780,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:04.769616+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:04.769616+0000","last_peered":"2026-03-31T21:21:04.769616+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:04.769616+0000","last_undegraded":"2026-03-31T21:21:04.769616+0000","last_fullsized":"2026-03-31T21:21:04.769616+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23100,"num_bytes_recovered":189180616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3754,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23100,"num_bytes_recovered":189180616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203612160,"data_stored":203612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104832,"kb_used_data":39216,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94267008,"statfs":{"total":96636764160,"available":96529416192,"internally_reserved":0,"allocated":40157184,"data_stored":39863510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39374848,"data_stored":39374848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:11.689 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:11.962 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:21:11.959+0000 7fe782ffd640 0 --2- 192.168.123.100:0/957626887 >> [v2:192.168.123.100:3300/0,v1:192.168.123.100:6789/0] conn(0x7fe78407fe30 0x7fe784129db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:21:12.077 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:12.077 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:12.087 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2026-03-31T21:21:11.947150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23150,"num_bytes_recovered":189902024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379180,"kb_used_data":147580,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736340,"statfs":{"total":289910292480,"available":289522012160,"internally_reserved":0,"allocated":151121920,"data_stored":150248066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":786456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001447"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59876,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:09.712798+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:09.712798+0000","last_peered":"2026-03-31T21:21:09.712798+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:09.712798+0000","last_undegraded":"2026-03-31T21:21:09.712798+0000","last_fullsized":"2026-03-31T21:21:09.712798+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203808768,"data_stored":203808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105024,"kb_used_data":39408,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266816,"statfs":{"total":96636764160,"available":96529219584,"internally_reserved":0,"allocated":40353792,"data_stored":40060118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39571456,"data_stored":39571456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:12.087 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:12.359 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:12.359 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:12.369 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2026-03-31T21:21:11.947150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23150,"num_bytes_recovered":189902024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379180,"kb_used_data":147580,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736340,"statfs":{"total":289910292480,"available":289522012160,"internally_reserved":0,"allocated":151121920,"data_stored":150248066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":786456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001447"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59876,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:09.712798+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:09.712798+0000","last_peered":"2026-03-31T21:21:09.712798+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:09.712798+0000","last_undegraded":"2026-03-31T21:21:09.712798+0000","last_fullsized":"2026-03-31T21:21:09.712798+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203808768,"data_stored":203808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105024,"kb_used_data":39408,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266816,"statfs":{"total":96636764160,"available":96529219584,"internally_reserved":0,"allocated":40353792,"data_stored":40060118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39571456,"data_stored":39571456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:15.370 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:15.553 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:15.553 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:15.564 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":660,"stamp":"2026-03-31T21:21:13.947406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23150,"num_bytes_recovered":189902024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379180,"kb_used_data":147580,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736340,"statfs":{"total":289910292480,"available":289522012160,"internally_reserved":0,"allocated":151121920,"data_stored":150248066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":786456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001467"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59876,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:09.712798+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:09.712798+0000","last_peered":"2026-03-31T21:21:09.712798+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:09.712798+0000","last_undegraded":"2026-03-31T21:21:09.712798+0000","last_fullsized":"2026-03-31T21:21:09.712798+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203808768,"data_stored":203808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105024,"kb_used_data":39408,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266816,"statfs":{"total":96636764160,"available":96529219584,"internally_reserved":0,"allocated":40353792,"data_stored":40060118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39571456,"data_stored":39571456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:15.564 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:15.747 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:15.748 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:15.757 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":660,"stamp":"2026-03-31T21:21:13.947406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23150,"num_bytes_recovered":189902024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379180,"kb_used_data":147580,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736340,"statfs":{"total":289910292480,"available":289522012160,"internally_reserved":0,"allocated":151121920,"data_stored":150248066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":786456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001467"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59876,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:09.712798+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:09.712798+0000","last_peered":"2026-03-31T21:21:09.712798+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:09.712798+0000","last_undegraded":"2026-03-31T21:21:09.712798+0000","last_fullsized":"2026-03-31T21:21:09.712798+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3706,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23148,"num_bytes_recovered":189573832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203808768,"data_stored":203808768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757594,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105024,"kb_used_data":39408,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266816,"statfs":{"total":96636764160,"available":96529219584,"internally_reserved":0,"allocated":40353792,"data_stored":40060118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39571456,"data_stored":39571456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:18.758 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:18.940 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:18.940 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:18.950 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2026-03-31T21:21:17.947883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3657,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23199,"num_bytes_recovered":190303432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379376,"kb_used_data":147776,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736144,"statfs":{"total":289910292480,"available":289521811456,"internally_reserved":0,"allocated":151322624,"data_stored":150448770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001475"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59974,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:14.718599+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:14.718599+0000","last_peered":"2026-03-31T21:21:14.718599+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:14.718599+0000","last_undegraded":"2026-03-31T21:21:14.718599+0000","last_fullsized":"2026-03-31T21:21:14.718599+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3657,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":189975240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3657,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":189975240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204009472,"data_stored":204009472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105220,"kb_used_data":39604,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266620,"statfs":{"total":96636764160,"available":96529018880,"internally_reserved":0,"allocated":40554496,"data_stored":40260822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39772160,"data_stored":39772160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:18.950 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:19.128 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:19.128 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:19.138 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":662,"stamp":"2026-03-31T21:21:17.947883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3657,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23199,"num_bytes_recovered":190303432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379376,"kb_used_data":147776,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282736144,"statfs":{"total":289910292480,"available":289521811456,"internally_reserved":0,"allocated":151322624,"data_stored":150448770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001475"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":59974,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:14.718599+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:14.718599+0000","last_peered":"2026-03-31T21:21:14.718599+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:14.718599+0000","last_undegraded":"2026-03-31T21:21:14.718599+0000","last_fullsized":"2026-03-31T21:21:14.718599+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3657,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":189975240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3657,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":189975240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204009472,"data_stored":204009472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267024,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757595,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105220,"kb_used_data":39604,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266620,"statfs":{"total":96636764160,"available":96529018880,"internally_reserved":0,"allocated":40554496,"data_stored":40260822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39772160,"data_stored":39772160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:22.139 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:22.319 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:22.319 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:22.328 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":664,"stamp":"2026-03-31T21:21:21.948395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23248,"num_bytes_recovered":190704840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379568,"kb_used_data":147968,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735952,"statfs":{"total":289910292480,"available":289521614848,"internally_reserved":0,"allocated":151519232,"data_stored":150645378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001492"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60072,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:19.747765+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:19.747765+0000","last_peered":"2026-03-31T21:21:19.747765+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:19.747765+0000","last_undegraded":"2026-03-31T21:21:19.747765+0000","last_fullsized":"2026-03-31T21:21:19.747765+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204210176,"data_stored":204210176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105412,"kb_used_data":39796,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266428,"statfs":{"total":96636764160,"available":96528822272,"internally_reserved":0,"allocated":40751104,"data_stored":40457430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39972864,"data_stored":39972864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:22.329 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:22.510 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:22.510 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:22.520 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":664,"stamp":"2026-03-31T21:21:21.948395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23248,"num_bytes_recovered":190704840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379568,"kb_used_data":147968,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735952,"statfs":{"total":289910292480,"available":289521614848,"internally_reserved":0,"allocated":151519232,"data_stored":150645378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001492"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60072,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:19.747765+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:19.747765+0000","last_peered":"2026-03-31T21:21:19.747765+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:19.747765+0000","last_undegraded":"2026-03-31T21:21:19.747765+0000","last_fullsized":"2026-03-31T21:21:19.747765+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204210176,"data_stored":204210176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105412,"kb_used_data":39796,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266428,"statfs":{"total":96636764160,"available":96528822272,"internally_reserved":0,"allocated":40751104,"data_stored":40457430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39972864,"data_stored":39972864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:25.522 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:25.710 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:25.711 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:25.721 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2026-03-31T21:21:23.948648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23248,"num_bytes_recovered":190704840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379568,"kb_used_data":147968,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735952,"statfs":{"total":289910292480,"available":289521614848,"internally_reserved":0,"allocated":151519232,"data_stored":150645378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60072,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:19.747765+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:19.747765+0000","last_peered":"2026-03-31T21:21:19.747765+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:19.747765+0000","last_undegraded":"2026-03-31T21:21:19.747765+0000","last_fullsized":"2026-03-31T21:21:19.747765+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204210176,"data_stored":204210176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105412,"kb_used_data":39796,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266428,"statfs":{"total":96636764160,"available":96528822272,"internally_reserved":0,"allocated":40751104,"data_stored":40457430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39972864,"data_stored":39972864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:25.721 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:25.901 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:25.901 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:25.911 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2026-03-31T21:21:23.948648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23248,"num_bytes_recovered":190704840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379568,"kb_used_data":147968,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735952,"statfs":{"total":289910292480,"available":289521614848,"internally_reserved":0,"allocated":151519232,"data_stored":150645378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001498"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60072,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:19.747765+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:19.747765+0000","last_peered":"2026-03-31T21:21:19.747765+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:19.747765+0000","last_undegraded":"2026-03-31T21:21:19.747765+0000","last_fullsized":"2026-03-31T21:21:19.747765+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3608,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23246,"num_bytes_recovered":190376648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204210176,"data_stored":204210176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430549,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757596,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105412,"kb_used_data":39796,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266428,"statfs":{"total":96636764160,"available":96528822272,"internally_reserved":0,"allocated":40751104,"data_stored":40457430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":39972864,"data_stored":39972864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:28.911 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:29.980 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:29.980 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:29.991 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2026-03-31T21:21:29.949421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23346,"num_bytes_recovered":191507656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379968,"kb_used_data":148368,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735552,"statfs":{"total":289910292480,"available":289521205248,"internally_reserved":0,"allocated":151928832,"data_stored":151054978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60268,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:29.767767+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:29.767767+0000","last_peered":"2026-03-31T21:21:29.767767+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:29.767767+0000","last_undegraded":"2026-03-31T21:21:29.767767+0000","last_fullsized":"2026-03-31T21:21:29.767767+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204611584,"data_stored":204611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105812,"kb_used_data":40196,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266028,"statfs":{"total":96636764160,"available":96528412672,"internally_reserved":0,"allocated":41160704,"data_stored":40867030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40374272,"data_stored":40374272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:29.991 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:30.172 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:30.172 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:30.183 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2026-03-31T21:21:29.949421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23346,"num_bytes_recovered":191507656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379968,"kb_used_data":148368,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735552,"statfs":{"total":289910292480,"available":289521205248,"internally_reserved":0,"allocated":151928832,"data_stored":151054978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60268,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:29.767767+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:29.767767+0000","last_peered":"2026-03-31T21:21:29.767767+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:29.767767+0000","last_undegraded":"2026-03-31T21:21:29.767767+0000","last_fullsized":"2026-03-31T21:21:29.767767+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204611584,"data_stored":204611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267026,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105812,"kb_used_data":40196,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266028,"statfs":{"total":96636764160,"available":96528412672,"internally_reserved":0,"allocated":41160704,"data_stored":40867030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40374272,"data_stored":40374272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:33.184 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:33.369 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:33.369 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:33.381 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":669,"stamp":"2026-03-31T21:21:31.949677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23346,"num_bytes_recovered":191507656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379968,"kb_used_data":148368,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735552,"statfs":{"total":289910292480,"available":289521205248,"internally_reserved":0,"allocated":151928832,"data_stored":151054978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60268,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:29.767767+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:29.767767+0000","last_peered":"2026-03-31T21:21:29.767767+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:29.767767+0000","last_undegraded":"2026-03-31T21:21:29.767767+0000","last_fullsized":"2026-03-31T21:21:29.767767+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204611584,"data_stored":204611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105812,"kb_used_data":40196,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266028,"statfs":{"total":96636764160,"available":96528412672,"internally_reserved":0,"allocated":41160704,"data_stored":40867030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40374272,"data_stored":40374272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:33.381 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:33.566 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:33.566 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:33.576 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":669,"stamp":"2026-03-31T21:21:31.949677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23346,"num_bytes_recovered":191507656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":379968,"kb_used_data":148368,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735552,"statfs":{"total":289910292480,"available":289521205248,"internally_reserved":0,"allocated":151928832,"data_stored":151054978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60268,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:29.767767+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:29.767767+0000","last_peered":"2026-03-31T21:21:29.767767+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:29.767767+0000","last_undegraded":"2026-03-31T21:21:29.767767+0000","last_fullsized":"2026-03-31T21:21:29.767767+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3510,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23344,"num_bytes_recovered":191179464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204611584,"data_stored":204611584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757598,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":105812,"kb_used_data":40196,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94266028,"statfs":{"total":96636764160,"available":96528412672,"internally_reserved":0,"allocated":41160704,"data_stored":40867030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40374272,"data_stored":40374272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:36.578 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:36.774 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:36.774 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:36.784 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2026-03-31T21:21:35.950131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3461,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23395,"num_bytes_recovered":191909064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380160,"kb_used_data":148560,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735360,"statfs":{"total":289910292480,"available":289521008640,"internally_reserved":0,"allocated":152125440,"data_stored":151251586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001482"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60366,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:34.782248+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:34.782248+0000","last_peered":"2026-03-31T21:21:34.782248+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:34.782248+0000","last_undegraded":"2026-03-31T21:21:34.782248+0000","last_fullsized":"2026-03-31T21:21:34.782248+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3461,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23393,"num_bytes_recovered":191580872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3461,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23393,"num_bytes_recovered":191580872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204812288,"data_stored":204812288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106004,"kb_used_data":40388,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265836,"statfs":{"total":96636764160,"available":96528216064,"internally_reserved":0,"allocated":41357312,"data_stored":41063638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40574976,"data_stored":40574976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:36.785 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:36.972 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:36.972 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:36.982 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":671,"stamp":"2026-03-31T21:21:35.950131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3461,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23395,"num_bytes_recovered":191909064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380160,"kb_used_data":148560,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735360,"statfs":{"total":289910292480,"available":289521008640,"internally_reserved":0,"allocated":152125440,"data_stored":151251586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001482"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60366,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:34.782248+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:34.782248+0000","last_peered":"2026-03-31T21:21:34.782248+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:34.782248+0000","last_undegraded":"2026-03-31T21:21:34.782248+0000","last_fullsized":"2026-03-31T21:21:34.782248+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3461,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23393,"num_bytes_recovered":191580872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":57,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:17:37.793918+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:17:37.793918+0000","last_peered":"2026-03-31T21:17:37.793918+0000","last_clean":"2026-03-31T21:17:37.793918+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:17:37.793918+0000","last_undegraded":"2026-03-31T21:17:37.793918+0000","last_fullsized":"2026-03-31T21:17:37.793918+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3461,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23393,"num_bytes_recovered":191580872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204812288,"data_stored":204812288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757599,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106004,"kb_used_data":40388,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265836,"statfs":{"total":96636764160,"available":96528216064,"internally_reserved":0,"allocated":41357312,"data_stored":41063638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40574976,"data_stored":40574976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:39.983 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:40.166 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:40.167 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:40.176 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2026-03-31T21:21:39.950648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23444,"num_bytes_recovered":192310472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380360,"kb_used_data":148760,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735160,"statfs":{"total":289910292480,"available":289520803840,"internally_reserved":0,"allocated":152330240,"data_stored":151456386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60464,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:39.792105+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:39.792105+0000","last_peered":"2026-03-31T21:21:39.792105+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:39.792105+0000","last_undegraded":"2026-03-31T21:21:39.792105+0000","last_fullsized":"2026-03-31T21:21:39.792105+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205012992,"data_stored":205012992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106204,"kb_used_data":40588,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265636,"statfs":{"total":96636764160,"available":96528011264,"internally_reserved":0,"allocated":41562112,"data_stored":41268438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40775680,"data_stored":40775680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:40.176 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:40.358 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:40.358 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:40.368 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":673,"stamp":"2026-03-31T21:21:39.950648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23444,"num_bytes_recovered":192310472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380360,"kb_used_data":148760,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735160,"statfs":{"total":289910292480,"available":289520803840,"internally_reserved":0,"allocated":152330240,"data_stored":151456386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60464,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:39.792105+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:39.792105+0000","last_peered":"2026-03-31T21:21:39.792105+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:39.792105+0000","last_undegraded":"2026-03-31T21:21:39.792105+0000","last_fullsized":"2026-03-31T21:21:39.792105+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205012992,"data_stored":205012992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267028,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106204,"kb_used_data":40588,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265636,"statfs":{"total":96636764160,"available":96528011264,"internally_reserved":0,"allocated":41562112,"data_stored":41268438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40775680,"data_stored":40775680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:43.369 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:43.554 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:43.554 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:43.564 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2026-03-31T21:21:41.950889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23444,"num_bytes_recovered":192310472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380360,"kb_used_data":148760,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735160,"statfs":{"total":289910292480,"available":289520803840,"internally_reserved":0,"allocated":152330240,"data_stored":151456386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60464,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:39.792105+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:39.792105+0000","last_peered":"2026-03-31T21:21:39.792105+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:39.792105+0000","last_undegraded":"2026-03-31T21:21:39.792105+0000","last_fullsized":"2026-03-31T21:21:39.792105+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205012992,"data_stored":205012992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106204,"kb_used_data":40588,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265636,"statfs":{"total":96636764160,"available":96528011264,"internally_reserved":0,"allocated":41562112,"data_stored":41268438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40775680,"data_stored":40775680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:43.564 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:43.747 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:43.748 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:43.757 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2026-03-31T21:21:41.950889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23444,"num_bytes_recovered":192310472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380360,"kb_used_data":148760,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282735160,"statfs":{"total":289910292480,"available":289520803840,"internally_reserved":0,"allocated":152330240,"data_stored":151456386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001468"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60464,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:39.792105+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:39.792105+0000","last_peered":"2026-03-31T21:21:39.792105+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:39.792105+0000","last_undegraded":"2026-03-31T21:21:39.792105+0000","last_fullsized":"2026-03-31T21:21:39.792105+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3412,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23442,"num_bytes_recovered":191982280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205012992,"data_stored":205012992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430553,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757600,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106204,"kb_used_data":40588,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265636,"statfs":{"total":96636764160,"available":96528011264,"internally_reserved":0,"allocated":41562112,"data_stored":41268438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40775680,"data_stored":40775680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:46.758 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:46.941 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:46.941 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:46.952 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":676,"stamp":"2026-03-31T21:21:45.951370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3364,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23492,"num_bytes_recovered":192703688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380556,"kb_used_data":148956,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734964,"statfs":{"total":289910292480,"available":289520603136,"internally_reserved":0,"allocated":152530944,"data_stored":151657090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001463"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60560,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:44.701011+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:44.701011+0000","last_peered":"2026-03-31T21:21:44.701011+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:44.701011+0000","last_undegraded":"2026-03-31T21:21:44.701011+0000","last_fullsized":"2026-03-31T21:21:44.701011+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3364,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23490,"num_bytes_recovered":192375496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3364,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23490,"num_bytes_recovered":192375496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205209600,"data_stored":205209600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430554,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313608,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106400,"kb_used_data":40784,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265440,"statfs":{"total":96636764160,"available":96527810560,"internally_reserved":0,"allocated":41762816,"data_stored":41469142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40972288,"data_stored":40972288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:46.952 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:47.134 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:47.135 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:47.144 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":676,"stamp":"2026-03-31T21:21:45.951370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3364,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23492,"num_bytes_recovered":192703688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380556,"kb_used_data":148956,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734964,"statfs":{"total":289910292480,"available":289520603136,"internally_reserved":0,"allocated":152530944,"data_stored":151657090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001463"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60560,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:44.701011+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:44.701011+0000","last_peered":"2026-03-31T21:21:44.701011+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:44.701011+0000","last_undegraded":"2026-03-31T21:21:44.701011+0000","last_fullsized":"2026-03-31T21:21:44.701011+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3364,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23490,"num_bytes_recovered":192375496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3364,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23490,"num_bytes_recovered":192375496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205209600,"data_stored":205209600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430554,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267029,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313608,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757601,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106400,"kb_used_data":40784,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265440,"statfs":{"total":96636764160,"available":96527810560,"internally_reserved":0,"allocated":41762816,"data_stored":41469142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40972288,"data_stored":40972288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:50.146 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:50.330 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:50.330 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:50.340 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2026-03-31T21:21:49.951893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23541,"num_bytes_recovered":193105096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380748,"kb_used_data":149148,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734772,"statfs":{"total":289910292480,"available":289520406528,"internally_reserved":0,"allocated":152727552,"data_stored":151853698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001508"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60658,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:49.727390+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:49.727390+0000","last_peered":"2026-03-31T21:21:49.727390+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:49.727390+0000","last_undegraded":"2026-03-31T21:21:49.727390+0000","last_fullsized":"2026-03-31T21:21:49.727390+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205410304,"data_stored":205410304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106592,"kb_used_data":40976,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265248,"statfs":{"total":96636764160,"available":96527613952,"internally_reserved":0,"allocated":41959424,"data_stored":41665750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41172992,"data_stored":41172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:50.340 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:50.529 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:50.530 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:50.540 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2026-03-31T21:21:49.951893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23541,"num_bytes_recovered":193105096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380748,"kb_used_data":149148,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734772,"statfs":{"total":289910292480,"available":289520406528,"internally_reserved":0,"allocated":152727552,"data_stored":151853698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001508"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60658,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:49.727390+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:49.727390+0000","last_peered":"2026-03-31T21:21:49.727390+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:49.727390+0000","last_undegraded":"2026-03-31T21:21:49.727390+0000","last_fullsized":"2026-03-31T21:21:49.727390+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205410304,"data_stored":205410304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106592,"kb_used_data":40976,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265248,"statfs":{"total":96636764160,"available":96527613952,"internally_reserved":0,"allocated":41959424,"data_stored":41665750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41172992,"data_stored":41172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:53.540 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:53.723 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:53.724 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:53.733 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2026-03-31T21:21:51.952127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23541,"num_bytes_recovered":193105096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380748,"kb_used_data":149148,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734772,"statfs":{"total":289910292480,"available":289520406528,"internally_reserved":0,"allocated":152727552,"data_stored":151853698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001478"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60658,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:49.727390+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:49.727390+0000","last_peered":"2026-03-31T21:21:49.727390+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:49.727390+0000","last_undegraded":"2026-03-31T21:21:49.727390+0000","last_fullsized":"2026-03-31T21:21:49.727390+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205410304,"data_stored":205410304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106592,"kb_used_data":40976,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265248,"statfs":{"total":96636764160,"available":96527613952,"internally_reserved":0,"allocated":41959424,"data_stored":41665750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41172992,"data_stored":41172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:53.734 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:53.917 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:53.917 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:53.928 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2026-03-31T21:21:51.952127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23541,"num_bytes_recovered":193105096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380748,"kb_used_data":149148,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734772,"statfs":{"total":289910292480,"available":289520406528,"internally_reserved":0,"allocated":152727552,"data_stored":151853698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001478"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60658,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:49.727390+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:49.727390+0000","last_peered":"2026-03-31T21:21:49.727390+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:49.727390+0000","last_undegraded":"2026-03-31T21:21:49.727390+0000","last_fullsized":"2026-03-31T21:21:49.727390+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23539,"num_bytes_recovered":192776904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205410304,"data_stored":205410304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430555,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757602,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106592,"kb_used_data":40976,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265248,"statfs":{"total":96636764160,"available":96527613952,"internally_reserved":0,"allocated":41959424,"data_stored":41665750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41172992,"data_stored":41172992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:56.929 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:57.116 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:57.117 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:57.127 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2026-03-31T21:21:55.952617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3266,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23590,"num_bytes_recovered":193506504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380944,"kb_used_data":149344,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734576,"statfs":{"total":289910292480,"available":289520205824,"internally_reserved":0,"allocated":152928256,"data_stored":152054402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60756,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:54.738737+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:54.738737+0000","last_peered":"2026-03-31T21:21:54.738737+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:54.738737+0000","last_undegraded":"2026-03-31T21:21:54.738737+0000","last_fullsized":"2026-03-31T21:21:54.738737+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23588,"num_bytes_recovered":193178312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23588,"num_bytes_recovered":193178312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205611008,"data_stored":205611008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313610,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106788,"kb_used_data":41172,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265052,"statfs":{"total":96636764160,"available":96527413248,"internally_reserved":0,"allocated":42160128,"data_stored":41866454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41373696,"data_stored":41373696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:21:57.128 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:21:57.322 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:21:57.322 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:21:57.333 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2026-03-31T21:21:55.952617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3266,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23590,"num_bytes_recovered":193506504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":380944,"kb_used_data":149344,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734576,"statfs":{"total":289910292480,"available":289520205824,"internally_reserved":0,"allocated":152928256,"data_stored":152054402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60756,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:54.738737+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:54.738737+0000","last_peered":"2026-03-31T21:21:54.738737+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:54.738737+0000","last_undegraded":"2026-03-31T21:21:54.738737+0000","last_fullsized":"2026-03-31T21:21:54.738737+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23588,"num_bytes_recovered":193178312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23588,"num_bytes_recovered":193178312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205611008,"data_stored":205611008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313610,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757603,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106788,"kb_used_data":41172,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94265052,"statfs":{"total":96636764160,"available":96527413248,"internally_reserved":0,"allocated":42160128,"data_stored":41866454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41373696,"data_stored":41373696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:00.334 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:00.518 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:00.518 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:00.528 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2026-03-31T21:21:59.953066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23639,"num_bytes_recovered":193907912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381144,"kb_used_data":149544,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734376,"statfs":{"total":289910292480,"available":289520001024,"internally_reserved":0,"allocated":153133056,"data_stored":152259202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001414"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60855,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:59.750422+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:59.750422+0000","last_peered":"2026-03-31T21:21:59.750422+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:59.750422+0000","last_undegraded":"2026-03-31T21:21:59.750422+0000","last_fullsized":"2026-03-31T21:21:59.750422+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205811712,"data_stored":205811712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106988,"kb_used_data":41372,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264852,"statfs":{"total":96636764160,"available":96527208448,"internally_reserved":0,"allocated":42364928,"data_stored":42071254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41574400,"data_stored":41574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:00.528 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:00.710 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:00.710 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:00.719 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2026-03-31T21:21:59.953066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23639,"num_bytes_recovered":193907912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381144,"kb_used_data":149544,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734376,"statfs":{"total":289910292480,"available":289520001024,"internally_reserved":0,"allocated":153133056,"data_stored":152259202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001414"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60855,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:59.750422+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:59.750422+0000","last_peered":"2026-03-31T21:21:59.750422+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:59.750422+0000","last_undegraded":"2026-03-31T21:21:59.750422+0000","last_fullsized":"2026-03-31T21:21:59.750422+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205811712,"data_stored":205811712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106988,"kb_used_data":41372,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264852,"statfs":{"total":96636764160,"available":96527208448,"internally_reserved":0,"allocated":42364928,"data_stored":42071254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41574400,"data_stored":41574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:03.721 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:03.900 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:03.901 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:03.910 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":684,"stamp":"2026-03-31T21:22:01.953314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23639,"num_bytes_recovered":193907912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381144,"kb_used_data":149544,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734376,"statfs":{"total":289910292480,"available":289520001024,"internally_reserved":0,"allocated":153133056,"data_stored":152259202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001420"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60855,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:59.750422+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:59.750422+0000","last_peered":"2026-03-31T21:21:59.750422+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:59.750422+0000","last_undegraded":"2026-03-31T21:21:59.750422+0000","last_fullsized":"2026-03-31T21:21:59.750422+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205811712,"data_stored":205811712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106988,"kb_used_data":41372,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264852,"statfs":{"total":96636764160,"available":96527208448,"internally_reserved":0,"allocated":42364928,"data_stored":42071254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41574400,"data_stored":41574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:03.910 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:04.090 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:04.090 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:04.100 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2026-03-31T21:22:03.953552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23639,"num_bytes_recovered":193907912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381144,"kb_used_data":149544,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734376,"statfs":{"total":289910292480,"available":289520001024,"internally_reserved":0,"allocated":153133056,"data_stored":152259202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60855,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:21:59.750422+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:21:59.750422+0000","last_peered":"2026-03-31T21:21:59.750422+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:21:59.750422+0000","last_undegraded":"2026-03-31T21:21:59.750422+0000","last_fullsized":"2026-03-31T21:21:59.750422+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23637,"num_bytes_recovered":193579720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":205811712,"data_stored":205811712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757604,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106988,"kb_used_data":41372,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264852,"statfs":{"total":96636764160,"available":96527208448,"internally_reserved":0,"allocated":42364928,"data_stored":42071254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41574400,"data_stored":41574400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:07.101 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:07.286 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:07.286 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:07.296 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2026-03-31T21:22:05.953805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3168,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23688,"num_bytes_recovered":194309320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381340,"kb_used_data":149740,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734180,"statfs":{"total":289910292480,"available":289519800320,"internally_reserved":0,"allocated":153333760,"data_stored":152459906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001422"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60953,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:04.757015+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:04.757015+0000","last_peered":"2026-03-31T21:22:04.757015+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:04.757015+0000","last_undegraded":"2026-03-31T21:22:04.757015+0000","last_fullsized":"2026-03-31T21:22:04.757015+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23686,"num_bytes_recovered":193981128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23686,"num_bytes_recovered":193981128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206012416,"data_stored":206012416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107184,"kb_used_data":41568,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264656,"statfs":{"total":96636764160,"available":96527007744,"internally_reserved":0,"allocated":42565632,"data_stored":42271958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41775104,"data_stored":41775104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:07.297 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:07.487 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:07.488 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:07.497 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2026-03-31T21:22:05.953805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3168,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23688,"num_bytes_recovered":194309320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381340,"kb_used_data":149740,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282734180,"statfs":{"total":289910292480,"available":289519800320,"internally_reserved":0,"allocated":153333760,"data_stored":152459906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001422"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":60953,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:04.757015+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:04.757015+0000","last_peered":"2026-03-31T21:22:04.757015+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:04.757015+0000","last_undegraded":"2026-03-31T21:22:04.757015+0000","last_fullsized":"2026-03-31T21:22:04.757015+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23686,"num_bytes_recovered":193981128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23686,"num_bytes_recovered":193981128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206012416,"data_stored":206012416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430558,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267034,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757605,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107184,"kb_used_data":41568,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264656,"statfs":{"total":96636764160,"available":96527007744,"internally_reserved":0,"allocated":42565632,"data_stored":42271958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41775104,"data_stored":41775104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:10.498 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:10.681 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:10.681 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:10.690 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2026-03-31T21:22:09.954294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23737,"num_bytes_recovered":194710728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381544,"kb_used_data":149944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733976,"statfs":{"total":289910292480,"available":289519591424,"internally_reserved":0,"allocated":153542656,"data_stored":152668802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001450"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61051,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:09.769948+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:09.769948+0000","last_peered":"2026-03-31T21:22:09.769948+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:09.769948+0000","last_undegraded":"2026-03-31T21:22:09.769948+0000","last_fullsized":"2026-03-31T21:22:09.769948+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206213120,"data_stored":206213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107388,"kb_used_data":41772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264452,"statfs":{"total":96636764160,"available":96526798848,"internally_reserved":0,"allocated":42774528,"data_stored":42480854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41975808,"data_stored":41975808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:10.691 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:10.873 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:10.873 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:10.883 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2026-03-31T21:22:09.954294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23737,"num_bytes_recovered":194710728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381544,"kb_used_data":149944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733976,"statfs":{"total":289910292480,"available":289519591424,"internally_reserved":0,"allocated":153542656,"data_stored":152668802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001450"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61051,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:09.769948+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:09.769948+0000","last_peered":"2026-03-31T21:22:09.769948+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:09.769948+0000","last_undegraded":"2026-03-31T21:22:09.769948+0000","last_fullsized":"2026-03-31T21:22:09.769948+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206213120,"data_stored":206213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107388,"kb_used_data":41772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264452,"statfs":{"total":96636764160,"available":96526798848,"internally_reserved":0,"allocated":42774528,"data_stored":42480854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41975808,"data_stored":41975808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:13.884 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:14.068 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:14.068 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:14.078 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2026-03-31T21:22:13.954773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23737,"num_bytes_recovered":194710728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381544,"kb_used_data":149944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733976,"statfs":{"total":289910292480,"available":289519591424,"internally_reserved":0,"allocated":153542656,"data_stored":152668802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001459"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61051,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:09.769948+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:09.769948+0000","last_peered":"2026-03-31T21:22:09.769948+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:09.769948+0000","last_undegraded":"2026-03-31T21:22:09.769948+0000","last_fullsized":"2026-03-31T21:22:09.769948+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206213120,"data_stored":206213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107388,"kb_used_data":41772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264452,"statfs":{"total":96636764160,"available":96526798848,"internally_reserved":0,"allocated":42774528,"data_stored":42480854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41975808,"data_stored":41975808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:14.078 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:14.257 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:14.257 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:14.267 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2026-03-31T21:22:13.954773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23737,"num_bytes_recovered":194710728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381544,"kb_used_data":149944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733976,"statfs":{"total":289910292480,"available":289519591424,"internally_reserved":0,"allocated":153542656,"data_stored":152668802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001459"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61051,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:09.769948+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:09.769948+0000","last_peered":"2026-03-31T21:22:09.769948+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:09.769948+0000","last_undegraded":"2026-03-31T21:22:09.769948+0000","last_fullsized":"2026-03-31T21:22:09.769948+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23735,"num_bytes_recovered":194382536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206213120,"data_stored":206213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430559,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757606,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107388,"kb_used_data":41772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264452,"statfs":{"total":96636764160,"available":96526798848,"internally_reserved":0,"allocated":42774528,"data_stored":42480854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41975808,"data_stored":41975808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:17.268 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:17.448 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:17.449 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:17.457 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":691,"stamp":"2026-03-31T21:22:15.954980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3070,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23786,"num_bytes_recovered":195112136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381740,"kb_used_data":150140,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733780,"statfs":{"total":289910292480,"available":289519390720,"internally_reserved":0,"allocated":153743360,"data_stored":152869506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001428"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61149,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:14.783316+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:14.783316+0000","last_peered":"2026-03-31T21:22:14.783316+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:14.783316+0000","last_undegraded":"2026-03-31T21:22:14.783316+0000","last_fullsized":"2026-03-31T21:22:14.783316+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23784,"num_bytes_recovered":194783944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23784,"num_bytes_recovered":194783944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206413824,"data_stored":206413824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430560,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107584,"kb_used_data":41968,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264256,"statfs":{"total":96636764160,"available":96526598144,"internally_reserved":0,"allocated":42975232,"data_stored":42681558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42176512,"data_stored":42176512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:17.457 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:17.636 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:17.637 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:17.647 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":691,"stamp":"2026-03-31T21:22:15.954980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3070,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23786,"num_bytes_recovered":195112136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381740,"kb_used_data":150140,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733780,"statfs":{"total":289910292480,"available":289519390720,"internally_reserved":0,"allocated":153743360,"data_stored":152869506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001428"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61149,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:14.783316+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:14.783316+0000","last_peered":"2026-03-31T21:22:14.783316+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:14.783316+0000","last_undegraded":"2026-03-31T21:22:14.783316+0000","last_fullsized":"2026-03-31T21:22:14.783316+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23784,"num_bytes_recovered":194783944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23784,"num_bytes_recovered":194783944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206413824,"data_stored":206413824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430560,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267036,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757607,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107584,"kb_used_data":41968,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264256,"statfs":{"total":96636764160,"available":96526598144,"internally_reserved":0,"allocated":42975232,"data_stored":42681558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42176512,"data_stored":42176512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:20.647 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:20.828 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:20.829 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:20.838 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":693,"stamp":"2026-03-31T21:22:19.955426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23835,"num_bytes_recovered":195513544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381936,"kb_used_data":150336,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733584,"statfs":{"total":289910292480,"available":289519190016,"internally_reserved":0,"allocated":153944064,"data_stored":153070210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001372"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61247,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:19.793173+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:19.793173+0000","last_peered":"2026-03-31T21:22:19.793173+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:19.793173+0000","last_undegraded":"2026-03-31T21:22:19.793173+0000","last_fullsized":"2026-03-31T21:22:19.793173+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206614528,"data_stored":206614528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430561,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757608,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107780,"kb_used_data":42164,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264060,"statfs":{"total":96636764160,"available":96526397440,"internally_reserved":0,"allocated":43175936,"data_stored":42882262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42377216,"data_stored":42377216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:20.838 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:21.017 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:21.017 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:21.028 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":693,"stamp":"2026-03-31T21:22:19.955426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23835,"num_bytes_recovered":195513544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381936,"kb_used_data":150336,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733584,"statfs":{"total":289910292480,"available":289519190016,"internally_reserved":0,"allocated":153944064,"data_stored":153070210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001372"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61247,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:19.793173+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:19.793173+0000","last_peered":"2026-03-31T21:22:19.793173+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:19.793173+0000","last_undegraded":"2026-03-31T21:22:19.793173+0000","last_fullsized":"2026-03-31T21:22:19.793173+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206614528,"data_stored":206614528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430561,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757608,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107780,"kb_used_data":42164,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264060,"statfs":{"total":96636764160,"available":96526397440,"internally_reserved":0,"allocated":43175936,"data_stored":42882262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42377216,"data_stored":42377216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:24.029 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:24.211 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:24.211 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:24.221 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2026-03-31T21:22:23.955920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23835,"num_bytes_recovered":195513544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381936,"kb_used_data":150336,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733584,"statfs":{"total":289910292480,"available":289519190016,"internally_reserved":0,"allocated":153944064,"data_stored":153070210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001367"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61247,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:19.793173+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:19.793173+0000","last_peered":"2026-03-31T21:22:19.793173+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:19.793173+0000","last_undegraded":"2026-03-31T21:22:19.793173+0000","last_fullsized":"2026-03-31T21:22:19.793173+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206614528,"data_stored":206614528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430561,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757608,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107780,"kb_used_data":42164,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264060,"statfs":{"total":96636764160,"available":96526397440,"internally_reserved":0,"allocated":43175936,"data_stored":42882262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42377216,"data_stored":42377216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:24.221 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:24.403 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:24.403 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:24.413 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2026-03-31T21:22:23.955920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23835,"num_bytes_recovered":195513544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":381936,"kb_used_data":150336,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733584,"statfs":{"total":289910292480,"available":289519190016,"internally_reserved":0,"allocated":153944064,"data_stored":153070210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001367"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61247,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:19.793173+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:19.793173+0000","last_peered":"2026-03-31T21:22:19.793173+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:19.793173+0000","last_undegraded":"2026-03-31T21:22:19.793173+0000","last_fullsized":"2026-03-31T21:22:19.793173+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23833,"num_bytes_recovered":195185352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206614528,"data_stored":206614528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430561,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757608,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107780,"kb_used_data":42164,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94264060,"statfs":{"total":96636764160,"available":96526397440,"internally_reserved":0,"allocated":43175936,"data_stored":42882262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42377216,"data_stored":42377216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:27.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:27.594 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:27.594 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:27.605 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2026-03-31T21:22:25.956161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2973,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23883,"num_bytes_recovered":195906760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382132,"kb_used_data":150532,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733388,"statfs":{"total":289910292480,"available":289518989312,"internally_reserved":0,"allocated":154144768,"data_stored":153270914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61343,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:24.704224+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:24.704224+0000","last_peered":"2026-03-31T21:22:24.704224+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:24.704224+0000","last_undegraded":"2026-03-31T21:22:24.704224+0000","last_fullsized":"2026-03-31T21:22:24.704224+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2973,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23881,"num_bytes_recovered":195578568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2973,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23881,"num_bytes_recovered":195578568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206811136,"data_stored":206811136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430562,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107976,"kb_used_data":42360,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263864,"statfs":{"total":96636764160,"available":96526196736,"internally_reserved":0,"allocated":43376640,"data_stored":43082966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42573824,"data_stored":42573824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:27.605 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:27.791 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:27.791 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:27.801 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2026-03-31T21:22:25.956161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2973,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23883,"num_bytes_recovered":195906760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382132,"kb_used_data":150532,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733388,"statfs":{"total":289910292480,"available":289518989312,"internally_reserved":0,"allocated":154144768,"data_stored":153270914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001387"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61343,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:24.704224+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:24.704224+0000","last_peered":"2026-03-31T21:22:24.704224+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:24.704224+0000","last_undegraded":"2026-03-31T21:22:24.704224+0000","last_fullsized":"2026-03-31T21:22:24.704224+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2973,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23881,"num_bytes_recovered":195578568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2973,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23881,"num_bytes_recovered":195578568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206811136,"data_stored":206811136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430562,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267038,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757609,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107976,"kb_used_data":42360,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263864,"statfs":{"total":96636764160,"available":96526196736,"internally_reserved":0,"allocated":43376640,"data_stored":43082966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42573824,"data_stored":42573824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:30.802 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:30.983 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:30.983 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:30.993 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-03-31T21:22:29.956610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23932,"num_bytes_recovered":196308168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382328,"kb_used_data":150728,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733192,"statfs":{"total":289910292480,"available":289518788608,"internally_reserved":0,"allocated":154345472,"data_stored":153471618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001430"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61441,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:29.710382+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:29.710382+0000","last_peered":"2026-03-31T21:22:29.710382+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:29.710382+0000","last_undegraded":"2026-03-31T21:22:29.710382+0000","last_fullsized":"2026-03-31T21:22:29.710382+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207011840,"data_stored":207011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757610,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108172,"kb_used_data":42556,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263668,"statfs":{"total":96636764160,"available":96525996032,"internally_reserved":0,"allocated":43577344,"data_stored":43283670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42774528,"data_stored":42774528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:30.993 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:31.174 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:31.174 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:31.183 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2026-03-31T21:22:29.956610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23932,"num_bytes_recovered":196308168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382328,"kb_used_data":150728,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733192,"statfs":{"total":289910292480,"available":289518788608,"internally_reserved":0,"allocated":154345472,"data_stored":153471618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001430"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61441,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:29.710382+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:29.710382+0000","last_peered":"2026-03-31T21:22:29.710382+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:29.710382+0000","last_undegraded":"2026-03-31T21:22:29.710382+0000","last_fullsized":"2026-03-31T21:22:29.710382+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207011840,"data_stored":207011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757610,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108172,"kb_used_data":42556,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263668,"statfs":{"total":96636764160,"available":96525996032,"internally_reserved":0,"allocated":43577344,"data_stored":43283670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42774528,"data_stored":42774528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:34.184 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:34.372 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:34.372 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:34.383 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2026-03-31T21:22:33.957136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23932,"num_bytes_recovered":196308168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382328,"kb_used_data":150728,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733192,"statfs":{"total":289910292480,"available":289518788608,"internally_reserved":0,"allocated":154345472,"data_stored":153471618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001455"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61441,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:29.710382+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:29.710382+0000","last_peered":"2026-03-31T21:22:29.710382+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:29.710382+0000","last_undegraded":"2026-03-31T21:22:29.710382+0000","last_fullsized":"2026-03-31T21:22:29.710382+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207011840,"data_stored":207011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757610,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108172,"kb_used_data":42556,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263668,"statfs":{"total":96636764160,"available":96525996032,"internally_reserved":0,"allocated":43577344,"data_stored":43283670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42774528,"data_stored":42774528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:34.383 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:34.566 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:34.566 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:34.576 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2026-03-31T21:22:33.957136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23932,"num_bytes_recovered":196308168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382328,"kb_used_data":150728,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733192,"statfs":{"total":289910292480,"available":289518788608,"internally_reserved":0,"allocated":154345472,"data_stored":153471618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001455"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61441,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:29.710382+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:29.710382+0000","last_peered":"2026-03-31T21:22:29.710382+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:29.710382+0000","last_undegraded":"2026-03-31T21:22:29.710382+0000","last_fullsized":"2026-03-31T21:22:29.710382+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2924,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23930,"num_bytes_recovered":195979976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207011840,"data_stored":207011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430563,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757610,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108172,"kb_used_data":42556,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263668,"statfs":{"total":96636764160,"available":96525996032,"internally_reserved":0,"allocated":43577344,"data_stored":43283670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42774528,"data_stored":42774528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:37.577 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:37.759 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:37.759 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:37.769 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2026-03-31T21:22:35.957471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2875,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23981,"num_bytes_recovered":196709576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382516,"kb_used_data":150916,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733004,"statfs":{"total":289910292480,"available":289518596096,"internally_reserved":0,"allocated":154537984,"data_stored":153664130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61539,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:34.725136+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:34.725136+0000","last_peered":"2026-03-31T21:22:34.725136+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:34.725136+0000","last_undegraded":"2026-03-31T21:22:34.725136+0000","last_fullsized":"2026-03-31T21:22:34.725136+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2875,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23979,"num_bytes_recovered":196381384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2875,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23979,"num_bytes_recovered":196381384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207212544,"data_stored":207212544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108360,"kb_used_data":42744,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263480,"statfs":{"total":96636764160,"available":96525803520,"internally_reserved":0,"allocated":43769856,"data_stored":43476182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42975232,"data_stored":42975232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:37.769 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:37.948 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:37.948 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:37.958 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2026-03-31T21:22:35.957471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2875,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23981,"num_bytes_recovered":196709576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382516,"kb_used_data":150916,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282733004,"statfs":{"total":289910292480,"available":289518596096,"internally_reserved":0,"allocated":154537984,"data_stored":153664130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61539,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:34.725136+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:34.725136+0000","last_peered":"2026-03-31T21:22:34.725136+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:34.725136+0000","last_undegraded":"2026-03-31T21:22:34.725136+0000","last_fullsized":"2026-03-31T21:22:34.725136+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2875,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23979,"num_bytes_recovered":196381384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2875,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23979,"num_bytes_recovered":196381384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207212544,"data_stored":207212544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430564,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267040,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757611,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108360,"kb_used_data":42744,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263480,"statfs":{"total":96636764160,"available":96525803520,"internally_reserved":0,"allocated":43769856,"data_stored":43476182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":42975232,"data_stored":42975232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:40.958 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:41.139 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:41.139 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:41.149 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2026-03-31T21:22:39.957938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24030,"num_bytes_recovered":197110984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382708,"kb_used_data":151108,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732812,"statfs":{"total":289910292480,"available":289518399488,"internally_reserved":0,"allocated":154734592,"data_stored":153860738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001553"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61637,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:39.729159+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:39.729159+0000","last_peered":"2026-03-31T21:22:39.729159+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:39.729159+0000","last_undegraded":"2026-03-31T21:22:39.729159+0000","last_fullsized":"2026-03-31T21:22:39.729159+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207413248,"data_stored":207413248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108552,"kb_used_data":42936,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263288,"statfs":{"total":96636764160,"available":96525606912,"internally_reserved":0,"allocated":43966464,"data_stored":43672790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43175936,"data_stored":43175936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:41.149 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:41.330 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:41.330 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:41.339 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2026-03-31T21:22:39.957938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24030,"num_bytes_recovered":197110984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382708,"kb_used_data":151108,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732812,"statfs":{"total":289910292480,"available":289518399488,"internally_reserved":0,"allocated":154734592,"data_stored":153860738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001553"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61637,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:39.729159+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:39.729159+0000","last_peered":"2026-03-31T21:22:39.729159+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:39.729159+0000","last_undegraded":"2026-03-31T21:22:39.729159+0000","last_fullsized":"2026-03-31T21:22:39.729159+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207413248,"data_stored":207413248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267041,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108552,"kb_used_data":42936,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263288,"statfs":{"total":96636764160,"available":96525606912,"internally_reserved":0,"allocated":43966464,"data_stored":43672790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43175936,"data_stored":43175936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:44.340 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:44.545 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:44.545 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:44.556 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":705,"stamp":"2026-03-31T21:22:43.958411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24030,"num_bytes_recovered":197110984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382708,"kb_used_data":151108,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732812,"statfs":{"total":289910292480,"available":289518399488,"internally_reserved":0,"allocated":154734592,"data_stored":153860738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001539"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61637,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:39.729159+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:39.729159+0000","last_peered":"2026-03-31T21:22:39.729159+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:39.729159+0000","last_undegraded":"2026-03-31T21:22:39.729159+0000","last_fullsized":"2026-03-31T21:22:39.729159+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207413248,"data_stored":207413248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108552,"kb_used_data":42936,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263288,"statfs":{"total":96636764160,"available":96525606912,"internally_reserved":0,"allocated":43966464,"data_stored":43672790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43175936,"data_stored":43175936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:44.556 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:44.742 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:44.742 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:44.752 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":705,"stamp":"2026-03-31T21:22:43.958411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24030,"num_bytes_recovered":197110984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382708,"kb_used_data":151108,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732812,"statfs":{"total":289910292480,"available":289518399488,"internally_reserved":0,"allocated":154734592,"data_stored":153860738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001539"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61637,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:39.729159+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:39.729159+0000","last_peered":"2026-03-31T21:22:39.729159+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:39.729159+0000","last_undegraded":"2026-03-31T21:22:39.729159+0000","last_fullsized":"2026-03-31T21:22:39.729159+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2826,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24028,"num_bytes_recovered":196782792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207413248,"data_stored":207413248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430565,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757612,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108552,"kb_used_data":42936,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263288,"statfs":{"total":96636764160,"available":96525606912,"internally_reserved":0,"allocated":43966464,"data_stored":43672790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43175936,"data_stored":43175936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:47.752 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:47.936 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:47.937 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:47.946 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2026-03-31T21:22:45.958661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2777,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24079,"num_bytes_recovered":197512392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382904,"kb_used_data":151304,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732616,"statfs":{"total":289910292480,"available":289518198784,"internally_reserved":0,"allocated":154935296,"data_stored":154061442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001525"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61735,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:44.738625+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:44.738625+0000","last_peered":"2026-03-31T21:22:44.738625+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:44.738625+0000","last_undegraded":"2026-03-31T21:22:44.738625+0000","last_fullsized":"2026-03-31T21:22:44.738625+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2777,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24077,"num_bytes_recovered":197184200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2777,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24077,"num_bytes_recovered":197184200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207613952,"data_stored":207613952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430566,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267042,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108748,"kb_used_data":43132,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263092,"statfs":{"total":96636764160,"available":96525406208,"internally_reserved":0,"allocated":44167168,"data_stored":43873494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43376640,"data_stored":43376640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:47.947 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:48.131 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:48.131 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:48.141 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2026-03-31T21:22:47.958900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2777,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24079,"num_bytes_recovered":197512392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":382904,"kb_used_data":151304,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732616,"statfs":{"total":289910292480,"available":289518198784,"internally_reserved":0,"allocated":154935296,"data_stored":154061442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001429"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61735,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:44.738625+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:44.738625+0000","last_peered":"2026-03-31T21:22:44.738625+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:44.738625+0000","last_undegraded":"2026-03-31T21:22:44.738625+0000","last_fullsized":"2026-03-31T21:22:44.738625+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2777,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24077,"num_bytes_recovered":197184200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2777,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24077,"num_bytes_recovered":197184200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207613952,"data_stored":207613952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430566,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757613,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108748,"kb_used_data":43132,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94263092,"statfs":{"total":96636764160,"available":96525406208,"internally_reserved":0,"allocated":44167168,"data_stored":43873494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43376640,"data_stored":43376640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:51.142 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:51.329 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:51.329 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:51.339 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":708,"stamp":"2026-03-31T21:22:49.959138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24126,"num_bytes_recovered":197897416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383096,"kb_used_data":151496,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732424,"statfs":{"total":289910292480,"available":289518002176,"internally_reserved":0,"allocated":155131904,"data_stored":154258050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61829,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:49.741939+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:49.741939+0000","last_peered":"2026-03-31T21:22:49.741939+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:49.741939+0000","last_undegraded":"2026-03-31T21:22:49.741939+0000","last_fullsized":"2026-03-31T21:22:49.741939+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207806464,"data_stored":207806464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430567,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108940,"kb_used_data":43324,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262900,"statfs":{"total":96636764160,"available":96525209600,"internally_reserved":0,"allocated":44363776,"data_stored":44070102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43569152,"data_stored":43569152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:51.339 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:51.532 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:51.532 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:51.543 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":708,"stamp":"2026-03-31T21:22:49.959138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24126,"num_bytes_recovered":197897416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383096,"kb_used_data":151496,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732424,"statfs":{"total":289910292480,"available":289518002176,"internally_reserved":0,"allocated":155131904,"data_stored":154258050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61829,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:49.741939+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:49.741939+0000","last_peered":"2026-03-31T21:22:49.741939+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:49.741939+0000","last_undegraded":"2026-03-31T21:22:49.741939+0000","last_fullsized":"2026-03-31T21:22:49.741939+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207806464,"data_stored":207806464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430567,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108940,"kb_used_data":43324,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262900,"statfs":{"total":96636764160,"available":96525209600,"internally_reserved":0,"allocated":44363776,"data_stored":44070102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43569152,"data_stored":43569152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:54.544 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:54.730 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:54.730 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:54.740 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2026-03-31T21:22:53.959629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24126,"num_bytes_recovered":197897416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383096,"kb_used_data":151496,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732424,"statfs":{"total":289910292480,"available":289518002176,"internally_reserved":0,"allocated":155131904,"data_stored":154258050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001465"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61829,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:49.741939+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:49.741939+0000","last_peered":"2026-03-31T21:22:49.741939+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:49.741939+0000","last_undegraded":"2026-03-31T21:22:49.741939+0000","last_fullsized":"2026-03-31T21:22:49.741939+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207806464,"data_stored":207806464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430567,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313623,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108940,"kb_used_data":43324,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262900,"statfs":{"total":96636764160,"available":96525209600,"internally_reserved":0,"allocated":44363776,"data_stored":44070102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43569152,"data_stored":43569152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:54.740 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:54.922 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:54.923 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:54.932 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2026-03-31T21:22:53.959629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24126,"num_bytes_recovered":197897416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383096,"kb_used_data":151496,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732424,"statfs":{"total":289910292480,"available":289518002176,"internally_reserved":0,"allocated":155131904,"data_stored":154258050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001465"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61829,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:49.741939+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:49.741939+0000","last_peered":"2026-03-31T21:22:49.741939+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:49.741939+0000","last_undegraded":"2026-03-31T21:22:49.741939+0000","last_fullsized":"2026-03-31T21:22:49.741939+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2730,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24124,"num_bytes_recovered":197569224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":207806464,"data_stored":207806464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430567,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313623,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757614,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108940,"kb_used_data":43324,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262900,"statfs":{"total":96636764160,"available":96525209600,"internally_reserved":0,"allocated":44363776,"data_stored":44070102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43569152,"data_stored":43569152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:57.934 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:58.117 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:58.117 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:58.127 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":712,"stamp":"2026-03-31T21:22:57.960068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2681,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24175,"num_bytes_recovered":198298824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383284,"kb_used_data":151684,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732236,"statfs":{"total":289910292480,"available":289517809664,"internally_reserved":0,"allocated":155324416,"data_stored":154450562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001406"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61927,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:54.753104+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:54.753104+0000","last_peered":"2026-03-31T21:22:54.753104+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:54.753104+0000","last_undegraded":"2026-03-31T21:22:54.753104+0000","last_fullsized":"2026-03-31T21:22:54.753104+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2681,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24173,"num_bytes_recovered":197970632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2681,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24173,"num_bytes_recovered":197970632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208007168,"data_stored":208007168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430568,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313624,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109128,"kb_used_data":43512,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262712,"statfs":{"total":96636764160,"available":96525017088,"internally_reserved":0,"allocated":44556288,"data_stored":44262614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43769856,"data_stored":43769856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:22:58.127 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:22:58.310 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:22:58.310 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:22:58.319 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":712,"stamp":"2026-03-31T21:22:57.960068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2681,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24175,"num_bytes_recovered":198298824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383284,"kb_used_data":151684,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732236,"statfs":{"total":289910292480,"available":289517809664,"internally_reserved":0,"allocated":155324416,"data_stored":154450562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001406"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":61927,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:54.753104+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:54.753104+0000","last_peered":"2026-03-31T21:22:54.753104+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:54.753104+0000","last_undegraded":"2026-03-31T21:22:54.753104+0000","last_fullsized":"2026-03-31T21:22:54.753104+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2681,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24173,"num_bytes_recovered":197970632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2681,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24173,"num_bytes_recovered":197970632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208007168,"data_stored":208007168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430568,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313624,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757615,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109128,"kb_used_data":43512,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262712,"statfs":{"total":96636764160,"available":96525017088,"internally_reserved":0,"allocated":44556288,"data_stored":44262614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43769856,"data_stored":43769856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:01.320 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:01.511 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:01.511 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:01.522 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2026-03-31T21:22:59.960315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24224,"num_bytes_recovered":198700232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383480,"kb_used_data":151880,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732040,"statfs":{"total":289910292480,"available":289517608960,"internally_reserved":0,"allocated":155525120,"data_stored":154651266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001414"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62025,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:59.761138+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:59.761138+0000","last_peered":"2026-03-31T21:22:59.761138+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:59.761138+0000","last_undegraded":"2026-03-31T21:22:59.761138+0000","last_fullsized":"2026-03-31T21:22:59.761138+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208207872,"data_stored":208207872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430570,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313624,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109324,"kb_used_data":43708,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262516,"statfs":{"total":96636764160,"available":96524816384,"internally_reserved":0,"allocated":44756992,"data_stored":44463318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43970560,"data_stored":43970560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:01.522 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:01.708 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:01.709 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:01.718 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2026-03-31T21:22:59.960315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24224,"num_bytes_recovered":198700232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383480,"kb_used_data":151880,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732040,"statfs":{"total":289910292480,"available":289517608960,"internally_reserved":0,"allocated":155525120,"data_stored":154651266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001414"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62025,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:59.761138+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:59.761138+0000","last_peered":"2026-03-31T21:22:59.761138+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:59.761138+0000","last_undegraded":"2026-03-31T21:22:59.761138+0000","last_fullsized":"2026-03-31T21:22:59.761138+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208207872,"data_stored":208207872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430570,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267045,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313624,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109324,"kb_used_data":43708,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262516,"statfs":{"total":96636764160,"available":96524816384,"internally_reserved":0,"allocated":44756992,"data_stored":44463318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43970560,"data_stored":43970560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:04.718 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:04.904 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:04.904 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:04.914 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2026-03-31T21:23:03.960752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24224,"num_bytes_recovered":198700232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383480,"kb_used_data":151880,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732040,"statfs":{"total":289910292480,"available":289517608960,"internally_reserved":0,"allocated":155525120,"data_stored":154651266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001356"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62025,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:59.761138+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:59.761138+0000","last_peered":"2026-03-31T21:22:59.761138+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:59.761138+0000","last_undegraded":"2026-03-31T21:22:59.761138+0000","last_fullsized":"2026-03-31T21:22:59.761138+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208207872,"data_stored":208207872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430570,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313625,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109324,"kb_used_data":43708,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262516,"statfs":{"total":96636764160,"available":96524816384,"internally_reserved":0,"allocated":44756992,"data_stored":44463318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43970560,"data_stored":43970560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:04.915 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:05.098 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:05.098 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:05.112 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2026-03-31T21:23:03.960752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24224,"num_bytes_recovered":198700232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383480,"kb_used_data":151880,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282732040,"statfs":{"total":289910292480,"available":289517608960,"internally_reserved":0,"allocated":155525120,"data_stored":154651266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001356"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62025,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:22:59.761138+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:22:59.761138+0000","last_peered":"2026-03-31T21:22:59.761138+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:22:59.761138+0000","last_undegraded":"2026-03-31T21:22:59.761138+0000","last_fullsized":"2026-03-31T21:22:59.761138+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2632,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24222,"num_bytes_recovered":198372040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208207872,"data_stored":208207872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430570,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267046,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313625,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757616,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109324,"kb_used_data":43708,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262516,"statfs":{"total":96636764160,"available":96524816384,"internally_reserved":0,"allocated":44756992,"data_stored":44463318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":43970560,"data_stored":43970560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:08.113 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:08.292 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:08.293 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:08.302 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2026-03-31T21:23:07.961282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2583,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24273,"num_bytes_recovered":199101640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383676,"kb_used_data":152076,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731844,"statfs":{"total":289910292480,"available":289517408256,"internally_reserved":0,"allocated":155725824,"data_stored":154851970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001432"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62123,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:04.756465+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:04.756465+0000","last_peered":"2026-03-31T21:23:04.756465+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:04.756465+0000","last_undegraded":"2026-03-31T21:23:04.756465+0000","last_fullsized":"2026-03-31T21:23:04.756465+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2583,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24271,"num_bytes_recovered":198773448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2583,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24271,"num_bytes_recovered":198773448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208408576,"data_stored":208408576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430571,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313626,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109520,"kb_used_data":43904,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262320,"statfs":{"total":96636764160,"available":96524615680,"internally_reserved":0,"allocated":44957696,"data_stored":44664022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44171264,"data_stored":44171264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:08.302 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:08.483 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:08.484 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:08.493 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2026-03-31T21:23:07.961282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2583,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24273,"num_bytes_recovered":199101640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383676,"kb_used_data":152076,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731844,"statfs":{"total":289910292480,"available":289517408256,"internally_reserved":0,"allocated":155725824,"data_stored":154851970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001432"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62123,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:04.756465+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:04.756465+0000","last_peered":"2026-03-31T21:23:04.756465+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:04.756465+0000","last_undegraded":"2026-03-31T21:23:04.756465+0000","last_fullsized":"2026-03-31T21:23:04.756465+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2583,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24271,"num_bytes_recovered":198773448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2583,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24271,"num_bytes_recovered":198773448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208408576,"data_stored":208408576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430571,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313626,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757617,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109520,"kb_used_data":43904,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262320,"statfs":{"total":96636764160,"available":96524615680,"internally_reserved":0,"allocated":44957696,"data_stored":44664022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44171264,"data_stored":44171264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:11.494 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:11.764 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:23:11.762+0000 7f090581f640 0 --2- 192.168.123.100:0/1395056609 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f090007f8c0 0x7f09001230a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:23:11.882 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:11.883 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:11.894 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":718,"stamp":"2026-03-31T21:23:09.961503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24322,"num_bytes_recovered":199503048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383872,"kb_used_data":152272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731648,"statfs":{"total":289910292480,"available":289517207552,"internally_reserved":0,"allocated":155926528,"data_stored":155052674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001435"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62221,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:09.751365+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:09.751365+0000","last_peered":"2026-03-31T21:23:09.751365+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:09.751365+0000","last_undegraded":"2026-03-31T21:23:09.751365+0000","last_fullsized":"2026-03-31T21:23:09.751365+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208609280,"data_stored":208609280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430572,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267047,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313626,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109716,"kb_used_data":44100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262124,"statfs":{"total":96636764160,"available":96524414976,"internally_reserved":0,"allocated":45158400,"data_stored":44864726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44371968,"data_stored":44371968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:11.894 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:12.076 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:12.076 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:12.087 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":719,"stamp":"2026-03-31T21:23:11.961732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24322,"num_bytes_recovered":199503048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383872,"kb_used_data":152272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731648,"statfs":{"total":289910292480,"available":289517207552,"internally_reserved":0,"allocated":155926528,"data_stored":155052674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001417"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62221,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:09.751365+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:09.751365+0000","last_peered":"2026-03-31T21:23:09.751365+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:09.751365+0000","last_undegraded":"2026-03-31T21:23:09.751365+0000","last_fullsized":"2026-03-31T21:23:09.751365+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208609280,"data_stored":208609280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430572,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313626,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109716,"kb_used_data":44100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262124,"statfs":{"total":96636764160,"available":96524414976,"internally_reserved":0,"allocated":45158400,"data_stored":44864726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44371968,"data_stored":44371968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:15.088 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:15.274 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:15.274 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:15.284 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":720,"stamp":"2026-03-31T21:23:13.962001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24322,"num_bytes_recovered":199503048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383872,"kb_used_data":152272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731648,"statfs":{"total":289910292480,"available":289517207552,"internally_reserved":0,"allocated":155926528,"data_stored":155052674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62221,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:09.751365+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:09.751365+0000","last_peered":"2026-03-31T21:23:09.751365+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:09.751365+0000","last_undegraded":"2026-03-31T21:23:09.751365+0000","last_fullsized":"2026-03-31T21:23:09.751365+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208609280,"data_stored":208609280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430572,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109716,"kb_used_data":44100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262124,"statfs":{"total":96636764160,"available":96524414976,"internally_reserved":0,"allocated":45158400,"data_stored":44864726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44371968,"data_stored":44371968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:15.284 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:15.466 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:15.466 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:15.476 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":720,"stamp":"2026-03-31T21:23:13.962001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24322,"num_bytes_recovered":199503048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":383872,"kb_used_data":152272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731648,"statfs":{"total":289910292480,"available":289517207552,"internally_reserved":0,"allocated":155926528,"data_stored":155052674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001480"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62221,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:09.751365+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:09.751365+0000","last_peered":"2026-03-31T21:23:09.751365+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:09.751365+0000","last_undegraded":"2026-03-31T21:23:09.751365+0000","last_fullsized":"2026-03-31T21:23:09.751365+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2534,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24320,"num_bytes_recovered":199174856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208609280,"data_stored":208609280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430572,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757618,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109716,"kb_used_data":44100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94262124,"statfs":{"total":96636764160,"available":96524414976,"internally_reserved":0,"allocated":45158400,"data_stored":44864726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44371968,"data_stored":44371968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:18.476 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:18.656 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:18.657 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:18.666 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2026-03-31T21:23:17.962508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2487,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24369,"num_bytes_recovered":199888072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384064,"kb_used_data":152464,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731456,"statfs":{"total":289910292480,"available":289517010944,"internally_reserved":0,"allocated":156123136,"data_stored":155249282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62315,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:14.744779+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:14.744779+0000","last_peered":"2026-03-31T21:23:14.744779+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:14.744779+0000","last_undegraded":"2026-03-31T21:23:14.744779+0000","last_fullsized":"2026-03-31T21:23:14.744779+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2487,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":199559880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2487,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":199559880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208801792,"data_stored":208801792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430573,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109908,"kb_used_data":44292,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261932,"statfs":{"total":96636764160,"available":96524218368,"internally_reserved":0,"allocated":45355008,"data_stored":45061334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44564480,"data_stored":44564480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:18.667 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:18.849 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:18.849 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:18.859 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2026-03-31T21:23:17.962508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2487,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24369,"num_bytes_recovered":199888072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384064,"kb_used_data":152464,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731456,"statfs":{"total":289910292480,"available":289517010944,"internally_reserved":0,"allocated":156123136,"data_stored":155249282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001485"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62315,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:14.744779+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:14.744779+0000","last_peered":"2026-03-31T21:23:14.744779+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:14.744779+0000","last_undegraded":"2026-03-31T21:23:14.744779+0000","last_fullsized":"2026-03-31T21:23:14.744779+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2487,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":199559880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2487,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24367,"num_bytes_recovered":199559880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":208801792,"data_stored":208801792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430573,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757619,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109908,"kb_used_data":44292,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261932,"statfs":{"total":96636764160,"available":96524218368,"internally_reserved":0,"allocated":45355008,"data_stored":45061334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44564480,"data_stored":44564480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:21.860 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:22.048 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:22.048 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:22.058 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2026-03-31T21:23:21.963004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24418,"num_bytes_recovered":200289480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384260,"kb_used_data":152660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731260,"statfs":{"total":289910292480,"available":289516810240,"internally_reserved":0,"allocated":156323840,"data_stored":155449986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62413,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:19.737189+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:19.737189+0000","last_peered":"2026-03-31T21:23:19.737189+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:19.737189+0000","last_undegraded":"2026-03-31T21:23:19.737189+0000","last_fullsized":"2026-03-31T21:23:19.737189+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209002496,"data_stored":209002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430574,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110104,"kb_used_data":44488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261736,"statfs":{"total":96636764160,"available":96524017664,"internally_reserved":0,"allocated":45555712,"data_stored":45262038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44765184,"data_stored":44765184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:22.058 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:22.243 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:22.243 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:22.254 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2026-03-31T21:23:21.963004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24418,"num_bytes_recovered":200289480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384260,"kb_used_data":152660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731260,"statfs":{"total":289910292480,"available":289516810240,"internally_reserved":0,"allocated":156323840,"data_stored":155449986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001500"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62413,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:19.737189+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:19.737189+0000","last_peered":"2026-03-31T21:23:19.737189+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:19.737189+0000","last_undegraded":"2026-03-31T21:23:19.737189+0000","last_fullsized":"2026-03-31T21:23:19.737189+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209002496,"data_stored":209002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430574,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110104,"kb_used_data":44488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261736,"statfs":{"total":96636764160,"available":96524017664,"internally_reserved":0,"allocated":45555712,"data_stored":45262038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44765184,"data_stored":44765184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:25.255 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:25.436 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:25.436 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:25.446 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2026-03-31T21:23:23.963252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24418,"num_bytes_recovered":200289480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384260,"kb_used_data":152660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731260,"statfs":{"total":289910292480,"available":289516810240,"internally_reserved":0,"allocated":156323840,"data_stored":155449986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62413,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:19.737189+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:19.737189+0000","last_peered":"2026-03-31T21:23:19.737189+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:19.737189+0000","last_undegraded":"2026-03-31T21:23:19.737189+0000","last_fullsized":"2026-03-31T21:23:19.737189+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209002496,"data_stored":209002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430574,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110104,"kb_used_data":44488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261736,"statfs":{"total":96636764160,"available":96524017664,"internally_reserved":0,"allocated":45555712,"data_stored":45262038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44765184,"data_stored":44765184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:25.446 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:25.626 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:25.626 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:25.635 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2026-03-31T21:23:23.963252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24418,"num_bytes_recovered":200289480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384260,"kb_used_data":152660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731260,"statfs":{"total":289910292480,"available":289516810240,"internally_reserved":0,"allocated":156323840,"data_stored":155449986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001519"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62413,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:19.737189+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:19.737189+0000","last_peered":"2026-03-31T21:23:19.737189+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:19.737189+0000","last_undegraded":"2026-03-31T21:23:19.737189+0000","last_fullsized":"2026-03-31T21:23:19.737189+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2438,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24416,"num_bytes_recovered":199961288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209002496,"data_stored":209002496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430574,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267050,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757620,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110104,"kb_used_data":44488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261736,"statfs":{"total":96636764160,"available":96524017664,"internally_reserved":0,"allocated":45555712,"data_stored":45262038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44765184,"data_stored":44765184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:28.637 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:28.817 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:28.817 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:28.827 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2026-03-31T21:23:27.963741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2389,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24467,"num_bytes_recovered":200690888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384460,"kb_used_data":152860,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731060,"statfs":{"total":289910292480,"available":289516605440,"internally_reserved":0,"allocated":156528640,"data_stored":155654786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001488"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62511,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:24.724434+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:24.724434+0000","last_peered":"2026-03-31T21:23:24.724434+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:24.724434+0000","last_undegraded":"2026-03-31T21:23:24.724434+0000","last_fullsized":"2026-03-31T21:23:24.724434+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2389,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24465,"num_bytes_recovered":200362696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2389,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24465,"num_bytes_recovered":200362696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209203200,"data_stored":209203200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430575,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313630,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757621,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110304,"kb_used_data":44688,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261536,"statfs":{"total":96636764160,"available":96523812864,"internally_reserved":0,"allocated":45760512,"data_stored":45466838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44965888,"data_stored":44965888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:28.828 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:29.008 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:29.008 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:29.017 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2026-03-31T21:23:27.963741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2389,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24467,"num_bytes_recovered":200690888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384460,"kb_used_data":152860,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282731060,"statfs":{"total":289910292480,"available":289516605440,"internally_reserved":0,"allocated":156528640,"data_stored":155654786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001488"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62511,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:24.724434+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:24.724434+0000","last_peered":"2026-03-31T21:23:24.724434+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:24.724434+0000","last_undegraded":"2026-03-31T21:23:24.724434+0000","last_fullsized":"2026-03-31T21:23:24.724434+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2389,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24465,"num_bytes_recovered":200362696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2389,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24465,"num_bytes_recovered":200362696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209203200,"data_stored":209203200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430575,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267051,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313630,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757621,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110304,"kb_used_data":44688,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261536,"statfs":{"total":96636764160,"available":96523812864,"internally_reserved":0,"allocated":45760512,"data_stored":45466838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44965888,"data_stored":44965888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:32.018 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:32.205 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:32.205 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:32.215 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2026-03-31T21:23:31.964267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24514,"num_bytes_recovered":201075912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384648,"kb_used_data":153048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730872,"statfs":{"total":289910292480,"available":289516412928,"internally_reserved":0,"allocated":156721152,"data_stored":155847298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62605,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:29.708163+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:29.708163+0000","last_peered":"2026-03-31T21:23:29.708163+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:29.708163+0000","last_undegraded":"2026-03-31T21:23:29.708163+0000","last_fullsized":"2026-03-31T21:23:29.708163+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209395712,"data_stored":209395712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430576,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313630,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110492,"kb_used_data":44876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261348,"statfs":{"total":96636764160,"available":96523620352,"internally_reserved":0,"allocated":45953024,"data_stored":45659350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45158400,"data_stored":45158400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:32.215 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:32.400 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:32.400 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:32.410 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2026-03-31T21:23:31.964267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24514,"num_bytes_recovered":201075912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384648,"kb_used_data":153048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730872,"statfs":{"total":289910292480,"available":289516412928,"internally_reserved":0,"allocated":156721152,"data_stored":155847298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62605,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:29.708163+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:29.708163+0000","last_peered":"2026-03-31T21:23:29.708163+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:29.708163+0000","last_undegraded":"2026-03-31T21:23:29.708163+0000","last_fullsized":"2026-03-31T21:23:29.708163+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209395712,"data_stored":209395712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430576,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313630,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110492,"kb_used_data":44876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261348,"statfs":{"total":96636764160,"available":96523620352,"internally_reserved":0,"allocated":45953024,"data_stored":45659350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45158400,"data_stored":45158400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:35.411 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:35.592 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:35.592 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:35.601 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2026-03-31T21:23:33.964513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24514,"num_bytes_recovered":201075912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384648,"kb_used_data":153048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730872,"statfs":{"total":289910292480,"available":289516412928,"internally_reserved":0,"allocated":156721152,"data_stored":155847298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62605,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:29.708163+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:29.708163+0000","last_peered":"2026-03-31T21:23:29.708163+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:29.708163+0000","last_undegraded":"2026-03-31T21:23:29.708163+0000","last_fullsized":"2026-03-31T21:23:29.708163+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209395712,"data_stored":209395712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430576,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313631,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110492,"kb_used_data":44876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261348,"statfs":{"total":96636764160,"available":96523620352,"internally_reserved":0,"allocated":45953024,"data_stored":45659350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45158400,"data_stored":45158400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:35.601 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:35.781 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:35.781 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:35.791 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2026-03-31T21:23:33.964513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24514,"num_bytes_recovered":201075912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384648,"kb_used_data":153048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730872,"statfs":{"total":289910292480,"available":289516412928,"internally_reserved":0,"allocated":156721152,"data_stored":155847298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62605,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:29.708163+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:29.708163+0000","last_peered":"2026-03-31T21:23:29.708163+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:29.708163+0000","last_undegraded":"2026-03-31T21:23:29.708163+0000","last_fullsized":"2026-03-31T21:23:29.708163+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":58,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:19:37.802135+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:19:37.802135+0000","last_peered":"2026-03-31T21:19:37.802135+0000","last_clean":"2026-03-31T21:19:37.802135+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:19:37.802135+0000","last_undegraded":"2026-03-31T21:19:37.802135+0000","last_fullsized":"2026-03-31T21:19:37.802135+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2342,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24512,"num_bytes_recovered":200747720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209395712,"data_stored":209395712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430576,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313631,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757622,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110492,"kb_used_data":44876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261348,"statfs":{"total":96636764160,"available":96523620352,"internally_reserved":0,"allocated":45953024,"data_stored":45659350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45158400,"data_stored":45158400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:38.791 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:38.974 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:38.974 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:38.984 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2026-03-31T21:23:37.965046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2292,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24564,"num_bytes_recovered":201485512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384844,"kb_used_data":153244,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730676,"statfs":{"total":289910292480,"available":289516212224,"internally_reserved":0,"allocated":156921856,"data_stored":156048002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62705,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:34.797749+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:34.797749+0000","last_peered":"2026-03-31T21:23:34.797749+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:34.797749+0000","last_undegraded":"2026-03-31T21:23:34.797749+0000","last_fullsized":"2026-03-31T21:23:34.797749+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2292,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24562,"num_bytes_recovered":201157320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2292,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24562,"num_bytes_recovered":201157320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209600512,"data_stored":209600512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430577,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757623,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110688,"kb_used_data":45072,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261152,"statfs":{"total":96636764160,"available":96523419648,"internally_reserved":0,"allocated":46153728,"data_stored":45860054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45363200,"data_stored":45363200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:38.984 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:39.169 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:39.169 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:39.179 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2026-03-31T21:23:37.965046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2292,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24564,"num_bytes_recovered":201485512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":384844,"kb_used_data":153244,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730676,"statfs":{"total":289910292480,"available":289516212224,"internally_reserved":0,"allocated":156921856,"data_stored":156048002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62705,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:34.797749+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:34.797749+0000","last_peered":"2026-03-31T21:23:34.797749+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:34.797749+0000","last_undegraded":"2026-03-31T21:23:34.797749+0000","last_fullsized":"2026-03-31T21:23:34.797749+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2292,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24562,"num_bytes_recovered":201157320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2292,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24562,"num_bytes_recovered":201157320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209600512,"data_stored":209600512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430577,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267053,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757623,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110688,"kb_used_data":45072,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94261152,"statfs":{"total":96636764160,"available":96523419648,"internally_reserved":0,"allocated":46153728,"data_stored":45860054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45363200,"data_stored":45363200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:42.180 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:42.447 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:23:42.445+0000 7f9380b70640 0 --2- 192.168.123.100:0/1281397939 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7f937c1756e0 0x7f937c195ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:23:42.563 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:42.563 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:42.573 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":734,"stamp":"2026-03-31T21:23:41.965573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24613,"num_bytes_recovered":201886920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385040,"kb_used_data":153440,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730480,"statfs":{"total":289910292480,"available":289516011520,"internally_reserved":0,"allocated":157122560,"data_stored":156248706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62803,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:39.784188+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:39.784188+0000","last_peered":"2026-03-31T21:23:39.784188+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:39.784188+0000","last_undegraded":"2026-03-31T21:23:39.784188+0000","last_fullsized":"2026-03-31T21:23:39.784188+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24611,"num_bytes_recovered":201558728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24611,"num_bytes_recovered":201558728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209801216,"data_stored":209801216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430578,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757624,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110884,"kb_used_data":45268,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260956,"statfs":{"total":96636764160,"available":96523218944,"internally_reserved":0,"allocated":46354432,"data_stored":46060758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45563904,"data_stored":45563904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:42.573 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:42.756 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:42.756 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:42.766 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":734,"stamp":"2026-03-31T21:23:41.965573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24613,"num_bytes_recovered":201886920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385040,"kb_used_data":153440,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730480,"statfs":{"total":289910292480,"available":289516011520,"internally_reserved":0,"allocated":157122560,"data_stored":156248706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62803,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:39.784188+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:39.784188+0000","last_peered":"2026-03-31T21:23:39.784188+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:39.784188+0000","last_undegraded":"2026-03-31T21:23:39.784188+0000","last_fullsized":"2026-03-31T21:23:39.784188+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24611,"num_bytes_recovered":201558728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24611,"num_bytes_recovered":201558728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209801216,"data_stored":209801216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430578,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757624,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110884,"kb_used_data":45268,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260956,"statfs":{"total":96636764160,"available":96523218944,"internally_reserved":0,"allocated":46354432,"data_stored":46060758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45563904,"data_stored":45563904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:45.767 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:45.950 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:45.950 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:45.960 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2026-03-31T21:23:43.965819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24613,"num_bytes_recovered":201886920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385040,"kb_used_data":153440,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730480,"statfs":{"total":289910292480,"available":289516011520,"internally_reserved":0,"allocated":157122560,"data_stored":156248706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001552"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62803,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:39.784188+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:39.784188+0000","last_peered":"2026-03-31T21:23:39.784188+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:39.784188+0000","last_undegraded":"2026-03-31T21:23:39.784188+0000","last_fullsized":"2026-03-31T21:23:39.784188+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24611,"num_bytes_recovered":201558728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2243,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24611,"num_bytes_recovered":201558728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":209801216,"data_stored":209801216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430578,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757624,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":110884,"kb_used_data":45268,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260956,"statfs":{"total":96636764160,"available":96523218944,"internally_reserved":0,"allocated":46354432,"data_stored":46060758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45563904,"data_stored":45563904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:45.960 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:46.152 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:46.152 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:46.163 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":736,"stamp":"2026-03-31T21:23:45.966070+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24662,"num_bytes_recovered":202288328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385228,"kb_used_data":153628,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730292,"statfs":{"total":289910292480,"available":289515819008,"internally_reserved":0,"allocated":157315072,"data_stored":156441218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-148,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62901,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:44.770706+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:44.770706+0000","last_peered":"2026-03-31T21:23:44.770706+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:44.770706+0000","last_undegraded":"2026-03-31T21:23:44.770706+0000","last_fullsized":"2026-03-31T21:23:44.770706+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24660,"num_bytes_recovered":201960136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24660,"num_bytes_recovered":201960136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210001920,"data_stored":210001920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430579,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267054,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757625,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111072,"kb_used_data":45456,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260768,"statfs":{"total":96636764160,"available":96523026432,"internally_reserved":0,"allocated":46546944,"data_stored":46253270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45764608,"data_stored":45764608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:49.164 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:49.347 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:49.348 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:49.358 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2026-03-31T21:23:47.966372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24662,"num_bytes_recovered":202288328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385228,"kb_used_data":153628,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730292,"statfs":{"total":289910292480,"available":289515819008,"internally_reserved":0,"allocated":157315072,"data_stored":156441218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62901,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:44.770706+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:44.770706+0000","last_peered":"2026-03-31T21:23:44.770706+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:44.770706+0000","last_undegraded":"2026-03-31T21:23:44.770706+0000","last_fullsized":"2026-03-31T21:23:44.770706+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24660,"num_bytes_recovered":201960136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24660,"num_bytes_recovered":201960136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210001920,"data_stored":210001920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430579,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757625,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111072,"kb_used_data":45456,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260768,"statfs":{"total":96636764160,"available":96523026432,"internally_reserved":0,"allocated":46546944,"data_stored":46253270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45764608,"data_stored":45764608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:49.358 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:49.540 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:49.540 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:49.550 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2026-03-31T21:23:47.966372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24662,"num_bytes_recovered":202288328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385228,"kb_used_data":153628,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730292,"statfs":{"total":289910292480,"available":289515819008,"internally_reserved":0,"allocated":157315072,"data_stored":156441218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001598"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62901,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:44.770706+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:44.770706+0000","last_peered":"2026-03-31T21:23:44.770706+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:44.770706+0000","last_undegraded":"2026-03-31T21:23:44.770706+0000","last_fullsized":"2026-03-31T21:23:44.770706+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24660,"num_bytes_recovered":201960136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2194,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24660,"num_bytes_recovered":201960136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210001920,"data_stored":210001920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430579,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757625,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111072,"kb_used_data":45456,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260768,"statfs":{"total":96636764160,"available":96523026432,"internally_reserved":0,"allocated":46546944,"data_stored":46253270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45764608,"data_stored":45764608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:52.550 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:52.739 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:52.740 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:52.749 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2026-03-31T21:23:51.966869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2145,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24711,"num_bytes_recovered":202689736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385424,"kb_used_data":153824,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730096,"statfs":{"total":289910292480,"available":289515618304,"internally_reserved":0,"allocated":157515776,"data_stored":156641922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62999,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:49.765470+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:49.765470+0000","last_peered":"2026-03-31T21:23:49.765470+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:49.765470+0000","last_undegraded":"2026-03-31T21:23:49.765470+0000","last_fullsized":"2026-03-31T21:23:49.765470+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2145,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24709,"num_bytes_recovered":202361544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2145,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24709,"num_bytes_recovered":202361544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210202624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757626,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111268,"kb_used_data":45652,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260572,"statfs":{"total":96636764160,"available":96522825728,"internally_reserved":0,"allocated":46747648,"data_stored":46453974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45965312,"data_stored":45965312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:52.750 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:52.930 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:52.930 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:52.940 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2026-03-31T21:23:51.966869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2145,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24711,"num_bytes_recovered":202689736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385424,"kb_used_data":153824,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282730096,"statfs":{"total":289910292480,"available":289515618304,"internally_reserved":0,"allocated":157515776,"data_stored":156641922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001568"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":62999,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:49.765470+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:49.765470+0000","last_peered":"2026-03-31T21:23:49.765470+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:49.765470+0000","last_undegraded":"2026-03-31T21:23:49.765470+0000","last_fullsized":"2026-03-31T21:23:49.765470+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2145,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24709,"num_bytes_recovered":202361544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2145,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24709,"num_bytes_recovered":202361544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210202624,"data_stored":210202624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430580,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757626,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111268,"kb_used_data":45652,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260572,"statfs":{"total":96636764160,"available":96522825728,"internally_reserved":0,"allocated":46747648,"data_stored":46453974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":45965312,"data_stored":45965312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:55.941 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:56.127 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:56.127 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:56.137 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":741,"stamp":"2026-03-31T21:23:55.967297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24760,"num_bytes_recovered":203091144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385616,"kb_used_data":154016,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729904,"statfs":{"total":289910292480,"available":289515421696,"internally_reserved":0,"allocated":157712384,"data_stored":156838530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63097,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:54.758203+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:54.758203+0000","last_peered":"2026-03-31T21:23:54.758203+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:54.758203+0000","last_undegraded":"2026-03-31T21:23:54.758203+0000","last_fullsized":"2026-03-31T21:23:54.758203+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210403328,"data_stored":210403328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430581,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313635,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111460,"kb_used_data":45844,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260380,"statfs":{"total":96636764160,"available":96522629120,"internally_reserved":0,"allocated":46944256,"data_stored":46650582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46166016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:56.137 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:56.324 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:56.324 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:56.334 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":741,"stamp":"2026-03-31T21:23:55.967297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24760,"num_bytes_recovered":203091144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385616,"kb_used_data":154016,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729904,"statfs":{"total":289910292480,"available":289515421696,"internally_reserved":0,"allocated":157712384,"data_stored":156838530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63097,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:54.758203+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:54.758203+0000","last_peered":"2026-03-31T21:23:54.758203+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:54.758203+0000","last_undegraded":"2026-03-31T21:23:54.758203+0000","last_fullsized":"2026-03-31T21:23:54.758203+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210403328,"data_stored":210403328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430581,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313635,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111460,"kb_used_data":45844,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260380,"statfs":{"total":96636764160,"available":96522629120,"internally_reserved":0,"allocated":46944256,"data_stored":46650582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46166016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:59.335 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:59.517 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:59.517 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:59.526 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2026-03-31T21:23:57.967558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24760,"num_bytes_recovered":203091144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385616,"kb_used_data":154016,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729904,"statfs":{"total":289910292480,"available":289515421696,"internally_reserved":0,"allocated":157712384,"data_stored":156838530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63097,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:54.758203+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:54.758203+0000","last_peered":"2026-03-31T21:23:54.758203+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:54.758203+0000","last_undegraded":"2026-03-31T21:23:54.758203+0000","last_fullsized":"2026-03-31T21:23:54.758203+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210403328,"data_stored":210403328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430581,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111460,"kb_used_data":45844,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260380,"statfs":{"total":96636764160,"available":96522629120,"internally_reserved":0,"allocated":46944256,"data_stored":46650582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46166016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:23:59.527 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:23:59.708 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:23:59.708 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:23:59.717 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2026-03-31T21:23:57.967558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24760,"num_bytes_recovered":203091144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385616,"kb_used_data":154016,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729904,"statfs":{"total":289910292480,"available":289515421696,"internally_reserved":0,"allocated":157712384,"data_stored":156838530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001487"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63097,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:54.758203+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:54.758203+0000","last_peered":"2026-03-31T21:23:54.758203+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:54.758203+0000","last_undegraded":"2026-03-31T21:23:54.758203+0000","last_fullsized":"2026-03-31T21:23:54.758203+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2096,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24758,"num_bytes_recovered":202762952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210403328,"data_stored":210403328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430581,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267057,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757627,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111460,"kb_used_data":45844,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260380,"statfs":{"total":96636764160,"available":96522629120,"internally_reserved":0,"allocated":46944256,"data_stored":46650582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46166016,"data_stored":46166016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:02.718 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:02.901 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:02.901 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:02.911 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2026-03-31T21:24:01.968029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2047,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24809,"num_bytes_recovered":203492552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385812,"kb_used_data":154212,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729708,"statfs":{"total":289910292480,"available":289515220992,"internally_reserved":0,"allocated":157913088,"data_stored":157039234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001373"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63196,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:59.750511+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:59.750511+0000","last_peered":"2026-03-31T21:23:59.750511+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:59.750511+0000","last_undegraded":"2026-03-31T21:23:59.750511+0000","last_fullsized":"2026-03-31T21:23:59.750511+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24807,"num_bytes_recovered":203164360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24807,"num_bytes_recovered":203164360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210604032,"data_stored":210604032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430582,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111656,"kb_used_data":46040,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260184,"statfs":{"total":96636764160,"available":96522428416,"internally_reserved":0,"allocated":47144960,"data_stored":46851286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46366720,"data_stored":46366720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:02.911 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:03.092 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:03.092 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:03.101 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2026-03-31T21:24:01.968029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2047,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24809,"num_bytes_recovered":203492552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":385812,"kb_used_data":154212,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729708,"statfs":{"total":289910292480,"available":289515220992,"internally_reserved":0,"allocated":157913088,"data_stored":157039234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001373"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63196,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:23:59.750511+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:23:59.750511+0000","last_peered":"2026-03-31T21:23:59.750511+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:23:59.750511+0000","last_undegraded":"2026-03-31T21:23:59.750511+0000","last_fullsized":"2026-03-31T21:23:59.750511+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24807,"num_bytes_recovered":203164360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2047,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24807,"num_bytes_recovered":203164360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210604032,"data_stored":210604032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430582,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757628,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111656,"kb_used_data":46040,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94260184,"statfs":{"total":96636764160,"available":96522428416,"internally_reserved":0,"allocated":47144960,"data_stored":46851286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46366720,"data_stored":46366720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:06.102 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:06.294 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:06.294 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:06.305 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2026-03-31T21:24:05.968525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24858,"num_bytes_recovered":203893960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386004,"kb_used_data":154404,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729516,"statfs":{"total":289910292480,"available":289515024384,"internally_reserved":0,"allocated":158109696,"data_stored":157235842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001443"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63294,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:04.741448+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:04.741448+0000","last_peered":"2026-03-31T21:24:04.741448+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:04.741448+0000","last_undegraded":"2026-03-31T21:24:04.741448+0000","last_fullsized":"2026-03-31T21:24:04.741448+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210804736,"data_stored":210804736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430583,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313637,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111848,"kb_used_data":46232,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259992,"statfs":{"total":96636764160,"available":96522231808,"internally_reserved":0,"allocated":47341568,"data_stored":47047894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46567424,"data_stored":46567424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:06.306 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:06.497 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:06.498 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:06.509 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2026-03-31T21:24:05.968525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24858,"num_bytes_recovered":203893960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386004,"kb_used_data":154404,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729516,"statfs":{"total":289910292480,"available":289515024384,"internally_reserved":0,"allocated":158109696,"data_stored":157235842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001443"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63294,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:04.741448+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:04.741448+0000","last_peered":"2026-03-31T21:24:04.741448+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:04.741448+0000","last_undegraded":"2026-03-31T21:24:04.741448+0000","last_fullsized":"2026-03-31T21:24:04.741448+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210804736,"data_stored":210804736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430583,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267058,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313637,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111848,"kb_used_data":46232,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259992,"statfs":{"total":96636764160,"available":96522231808,"internally_reserved":0,"allocated":47341568,"data_stored":47047894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46567424,"data_stored":46567424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:09.509 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:09.695 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:09.695 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:09.705 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":747,"stamp":"2026-03-31T21:24:07.968817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24858,"num_bytes_recovered":203893960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386004,"kb_used_data":154404,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729516,"statfs":{"total":289910292480,"available":289515024384,"internally_reserved":0,"allocated":158109696,"data_stored":157235842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001520"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63294,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:04.741448+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:04.741448+0000","last_peered":"2026-03-31T21:24:04.741448+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:04.741448+0000","last_undegraded":"2026-03-31T21:24:04.741448+0000","last_fullsized":"2026-03-31T21:24:04.741448+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210804736,"data_stored":210804736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430583,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111848,"kb_used_data":46232,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259992,"statfs":{"total":96636764160,"available":96522231808,"internally_reserved":0,"allocated":47341568,"data_stored":47047894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46567424,"data_stored":46567424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:09.705 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:09.888 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:09.888 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:09.898 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":747,"stamp":"2026-03-31T21:24:07.968817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24858,"num_bytes_recovered":203893960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386004,"kb_used_data":154404,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729516,"statfs":{"total":289910292480,"available":289515024384,"internally_reserved":0,"allocated":158109696,"data_stored":157235842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001520"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63294,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:04.741448+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:04.741448+0000","last_peered":"2026-03-31T21:24:04.741448+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:04.741448+0000","last_undegraded":"2026-03-31T21:24:04.741448+0000","last_fullsized":"2026-03-31T21:24:04.741448+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1998,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24856,"num_bytes_recovered":203565768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210804736,"data_stored":210804736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430583,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267059,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757629,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111848,"kb_used_data":46232,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259992,"statfs":{"total":96636764160,"available":96522231808,"internally_reserved":0,"allocated":47341568,"data_stored":47047894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46567424,"data_stored":46567424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:12.898 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:13.083 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:13.083 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:13.093 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2026-03-31T21:24:11.969313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1949,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24907,"num_bytes_recovered":204295368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386200,"kb_used_data":154600,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729320,"statfs":{"total":289910292480,"available":289514823680,"internally_reserved":0,"allocated":158310400,"data_stored":157436546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63392,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:09.741980+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:09.741980+0000","last_peered":"2026-03-31T21:24:09.741980+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:09.741980+0000","last_undegraded":"2026-03-31T21:24:09.741980+0000","last_fullsized":"2026-03-31T21:24:09.741980+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1949,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24905,"num_bytes_recovered":203967176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1949,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24905,"num_bytes_recovered":203967176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211005440,"data_stored":211005440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757630,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112044,"kb_used_data":46428,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259796,"statfs":{"total":96636764160,"available":96522031104,"internally_reserved":0,"allocated":47542272,"data_stored":47248598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46768128,"data_stored":46768128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:13.093 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:13.279 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:13.279 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:13.289 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2026-03-31T21:24:11.969313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1949,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24907,"num_bytes_recovered":204295368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386200,"kb_used_data":154600,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729320,"statfs":{"total":289910292480,"available":289514823680,"internally_reserved":0,"allocated":158310400,"data_stored":157436546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001502"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63392,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:09.741980+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:09.741980+0000","last_peered":"2026-03-31T21:24:09.741980+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:09.741980+0000","last_undegraded":"2026-03-31T21:24:09.741980+0000","last_fullsized":"2026-03-31T21:24:09.741980+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1949,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24905,"num_bytes_recovered":203967176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1949,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24905,"num_bytes_recovered":203967176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211005440,"data_stored":211005440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757630,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112044,"kb_used_data":46428,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259796,"statfs":{"total":96636764160,"available":96522031104,"internally_reserved":0,"allocated":47542272,"data_stored":47248598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46768128,"data_stored":46768128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:16.290 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:16.473 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:16.473 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:16.483 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2026-03-31T21:24:15.969794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24956,"num_bytes_recovered":204696776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386404,"kb_used_data":154804,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729116,"statfs":{"total":289910292480,"available":289514614784,"internally_reserved":0,"allocated":158519296,"data_stored":157645442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63490,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:14.735111+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:14.735111+0000","last_peered":"2026-03-31T21:24:14.735111+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:14.735111+0000","last_undegraded":"2026-03-31T21:24:14.735111+0000","last_fullsized":"2026-03-31T21:24:14.735111+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24954,"num_bytes_recovered":204368584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24954,"num_bytes_recovered":204368584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211206144,"data_stored":211206144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313639,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757631,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112248,"kb_used_data":46632,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259592,"statfs":{"total":96636764160,"available":96521822208,"internally_reserved":0,"allocated":47751168,"data_stored":47457494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46968832,"data_stored":46968832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:16.483 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:16.663 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:16.663 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:16.674 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2026-03-31T21:24:15.969794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24956,"num_bytes_recovered":204696776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386404,"kb_used_data":154804,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729116,"statfs":{"total":289910292480,"available":289514614784,"internally_reserved":0,"allocated":158519296,"data_stored":157645442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001514"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63490,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:14.735111+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:14.735111+0000","last_peered":"2026-03-31T21:24:14.735111+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:14.735111+0000","last_undegraded":"2026-03-31T21:24:14.735111+0000","last_fullsized":"2026-03-31T21:24:14.735111+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24954,"num_bytes_recovered":204368584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24954,"num_bytes_recovered":204368584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211206144,"data_stored":211206144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313639,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757631,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112248,"kb_used_data":46632,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259592,"statfs":{"total":96636764160,"available":96521822208,"internally_reserved":0,"allocated":47751168,"data_stored":47457494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46968832,"data_stored":46968832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:19.674 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:19.860 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:19.860 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:19.870 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":752,"stamp":"2026-03-31T21:24:17.970023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24956,"num_bytes_recovered":204696776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386404,"kb_used_data":154804,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282729116,"statfs":{"total":289910292480,"available":289514614784,"internally_reserved":0,"allocated":158519296,"data_stored":157645442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001497"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63490,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:14.735111+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:14.735111+0000","last_peered":"2026-03-31T21:24:14.735111+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:14.735111+0000","last_undegraded":"2026-03-31T21:24:14.735111+0000","last_fullsized":"2026-03-31T21:24:14.735111+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24954,"num_bytes_recovered":204368584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1900,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24954,"num_bytes_recovered":204368584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211206144,"data_stored":211206144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757631,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112248,"kb_used_data":46632,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259592,"statfs":{"total":96636764160,"available":96521822208,"internally_reserved":0,"allocated":47751168,"data_stored":47457494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46968832,"data_stored":46968832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:19.870 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:20.057 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:20.057 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:20.067 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2026-03-31T21:24:19.970241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25005,"num_bytes_recovered":205098184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386600,"kb_used_data":155000,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728920,"statfs":{"total":289910292480,"available":289514414080,"internally_reserved":0,"allocated":158720000,"data_stored":157846146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001424"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63588,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:19.722731+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:19.722731+0000","last_peered":"2026-03-31T21:24:19.722731+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:19.722731+0000","last_undegraded":"2026-03-31T21:24:19.722731+0000","last_fullsized":"2026-03-31T21:24:19.722731+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25003,"num_bytes_recovered":204769992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25003,"num_bytes_recovered":204769992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211406848,"data_stored":211406848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430586,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267061,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112444,"kb_used_data":46828,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259396,"statfs":{"total":96636764160,"available":96521621504,"internally_reserved":0,"allocated":47951872,"data_stored":47658198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47169536,"data_stored":47169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:23.068 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:23.252 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:23.252 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:23.263 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2026-03-31T21:24:21.970476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25005,"num_bytes_recovered":205098184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386600,"kb_used_data":155000,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728920,"statfs":{"total":289910292480,"available":289514414080,"internally_reserved":0,"allocated":158720000,"data_stored":157846146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63588,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:19.722731+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:19.722731+0000","last_peered":"2026-03-31T21:24:19.722731+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:19.722731+0000","last_undegraded":"2026-03-31T21:24:19.722731+0000","last_fullsized":"2026-03-31T21:24:19.722731+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25003,"num_bytes_recovered":204769992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25003,"num_bytes_recovered":204769992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211406848,"data_stored":211406848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430586,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112444,"kb_used_data":46828,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259396,"statfs":{"total":96636764160,"available":96521621504,"internally_reserved":0,"allocated":47951872,"data_stored":47658198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47169536,"data_stored":47169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:23.263 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:23.444 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:23.445 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:23.454 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":754,"stamp":"2026-03-31T21:24:21.970476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25005,"num_bytes_recovered":205098184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386600,"kb_used_data":155000,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728920,"statfs":{"total":289910292480,"available":289514414080,"internally_reserved":0,"allocated":158720000,"data_stored":157846146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001400"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63588,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:19.722731+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:19.722731+0000","last_peered":"2026-03-31T21:24:19.722731+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:19.722731+0000","last_undegraded":"2026-03-31T21:24:19.722731+0000","last_fullsized":"2026-03-31T21:24:19.722731+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25003,"num_bytes_recovered":204769992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1851,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25003,"num_bytes_recovered":204769992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211406848,"data_stored":211406848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430586,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757632,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112444,"kb_used_data":46828,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259396,"statfs":{"total":96636764160,"available":96521621504,"internally_reserved":0,"allocated":47951872,"data_stored":47658198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47169536,"data_stored":47169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:26.455 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:26.641 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:26.642 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:26.651 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":756,"stamp":"2026-03-31T21:24:25.970958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1802,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25054,"num_bytes_recovered":205499592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386796,"kb_used_data":155196,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728724,"statfs":{"total":289910292480,"available":289514213376,"internally_reserved":0,"allocated":158920704,"data_stored":158046850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001416"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63686,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:24.720416+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:24.720416+0000","last_peered":"2026-03-31T21:24:24.720416+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:24.720416+0000","last_undegraded":"2026-03-31T21:24:24.720416+0000","last_fullsized":"2026-03-31T21:24:24.720416+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1802,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25052,"num_bytes_recovered":205171400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1802,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25052,"num_bytes_recovered":205171400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211607552,"data_stored":211607552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430587,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313641,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112640,"kb_used_data":47024,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259200,"statfs":{"total":96636764160,"available":96521420800,"internally_reserved":0,"allocated":48152576,"data_stored":47858902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47370240,"data_stored":47370240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:26.652 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:26.835 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:26.835 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:26.844 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":756,"stamp":"2026-03-31T21:24:25.970958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1802,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25054,"num_bytes_recovered":205499592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386796,"kb_used_data":155196,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728724,"statfs":{"total":289910292480,"available":289514213376,"internally_reserved":0,"allocated":158920704,"data_stored":158046850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001416"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63686,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:24.720416+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:24.720416+0000","last_peered":"2026-03-31T21:24:24.720416+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:24.720416+0000","last_undegraded":"2026-03-31T21:24:24.720416+0000","last_fullsized":"2026-03-31T21:24:24.720416+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1802,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25052,"num_bytes_recovered":205171400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1802,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25052,"num_bytes_recovered":205171400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211607552,"data_stored":211607552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430587,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267062,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313641,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757633,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112640,"kb_used_data":47024,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259200,"statfs":{"total":96636764160,"available":96521420800,"internally_reserved":0,"allocated":48152576,"data_stored":47858902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47370240,"data_stored":47370240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:29.845 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:30.032 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:30.033 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:30.043 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":758,"stamp":"2026-03-31T21:24:29.971500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25103,"num_bytes_recovered":205901000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386992,"kb_used_data":155392,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728528,"statfs":{"total":289910292480,"available":289514012672,"internally_reserved":0,"allocated":159121408,"data_stored":158247554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63784,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:29.708419+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:29.708419+0000","last_peered":"2026-03-31T21:24:29.708419+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:29.708419+0000","last_undegraded":"2026-03-31T21:24:29.708419+0000","last_fullsized":"2026-03-31T21:24:29.708419+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211808256,"data_stored":211808256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430588,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112836,"kb_used_data":47220,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259004,"statfs":{"total":96636764160,"available":96521220096,"internally_reserved":0,"allocated":48353280,"data_stored":48059606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47570944,"data_stored":47570944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:30.044 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:30.232 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:30.232 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:30.242 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":758,"stamp":"2026-03-31T21:24:29.971500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25103,"num_bytes_recovered":205901000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386992,"kb_used_data":155392,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728528,"statfs":{"total":289910292480,"available":289514012672,"internally_reserved":0,"allocated":159121408,"data_stored":158247554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63784,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:29.708419+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:29.708419+0000","last_peered":"2026-03-31T21:24:29.708419+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:29.708419+0000","last_undegraded":"2026-03-31T21:24:29.708419+0000","last_fullsized":"2026-03-31T21:24:29.708419+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211808256,"data_stored":211808256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430588,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267063,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112836,"kb_used_data":47220,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259004,"statfs":{"total":96636764160,"available":96521220096,"internally_reserved":0,"allocated":48353280,"data_stored":48059606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47570944,"data_stored":47570944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:33.244 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:33.445 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:33.445 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:33.457 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2026-03-31T21:24:31.971753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25103,"num_bytes_recovered":205901000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386992,"kb_used_data":155392,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728528,"statfs":{"total":289910292480,"available":289514012672,"internally_reserved":0,"allocated":159121408,"data_stored":158247554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63784,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:29.708419+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:29.708419+0000","last_peered":"2026-03-31T21:24:29.708419+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:29.708419+0000","last_undegraded":"2026-03-31T21:24:29.708419+0000","last_fullsized":"2026-03-31T21:24:29.708419+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211808256,"data_stored":211808256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430588,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112836,"kb_used_data":47220,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259004,"statfs":{"total":96636764160,"available":96521220096,"internally_reserved":0,"allocated":48353280,"data_stored":48059606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47570944,"data_stored":47570944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:33.458 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:33.655 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:33.656 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:33.667 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2026-03-31T21:24:31.971753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25103,"num_bytes_recovered":205901000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":386992,"kb_used_data":155392,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728528,"statfs":{"total":289910292480,"available":289514012672,"internally_reserved":0,"allocated":159121408,"data_stored":158247554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001512"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63784,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:29.708419+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:29.708419+0000","last_peered":"2026-03-31T21:24:29.708419+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:29.708419+0000","last_undegraded":"2026-03-31T21:24:29.708419+0000","last_fullsized":"2026-03-31T21:24:29.708419+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1753,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25101,"num_bytes_recovered":205572808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":211808256,"data_stored":211808256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430588,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757634,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112836,"kb_used_data":47220,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94259004,"statfs":{"total":96636764160,"available":96521220096,"internally_reserved":0,"allocated":48353280,"data_stored":48059606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47570944,"data_stored":47570944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:36.668 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:36.863 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:36.863 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:36.875 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":761,"stamp":"2026-03-31T21:24:35.972305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1704,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25152,"num_bytes_recovered":206302408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387192,"kb_used_data":155592,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728328,"statfs":{"total":289910292480,"available":289513807872,"internally_reserved":0,"allocated":159326208,"data_stored":158452354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63882,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:34.710690+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:34.710690+0000","last_peered":"2026-03-31T21:24:34.710690+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:34.710690+0000","last_undegraded":"2026-03-31T21:24:34.710690+0000","last_fullsized":"2026-03-31T21:24:34.710690+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1704,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25150,"num_bytes_recovered":205974216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1704,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25150,"num_bytes_recovered":205974216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212008960,"data_stored":212008960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313643,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757635,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113036,"kb_used_data":47420,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258804,"statfs":{"total":96636764160,"available":96521015296,"internally_reserved":0,"allocated":48558080,"data_stored":48264406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47771648,"data_stored":47771648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:36.876 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:37.066 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:37.066 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:37.079 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":761,"stamp":"2026-03-31T21:24:35.972305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1704,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25152,"num_bytes_recovered":206302408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387192,"kb_used_data":155592,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728328,"statfs":{"total":289910292480,"available":289513807872,"internally_reserved":0,"allocated":159326208,"data_stored":158452354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001615"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63882,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:34.710690+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:34.710690+0000","last_peered":"2026-03-31T21:24:34.710690+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:34.710690+0000","last_undegraded":"2026-03-31T21:24:34.710690+0000","last_fullsized":"2026-03-31T21:24:34.710690+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1704,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25150,"num_bytes_recovered":205974216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1704,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25150,"num_bytes_recovered":205974216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212008960,"data_stored":212008960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313643,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757635,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113036,"kb_used_data":47420,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258804,"statfs":{"total":96636764160,"available":96521015296,"internally_reserved":0,"allocated":48558080,"data_stored":48264406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47771648,"data_stored":47771648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:40.080 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:40.276 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:40.276 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:40.288 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":763,"stamp":"2026-03-31T21:24:39.973003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25201,"num_bytes_recovered":206703816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387384,"kb_used_data":155784,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728136,"statfs":{"total":289910292480,"available":289513611264,"internally_reserved":0,"allocated":159522816,"data_stored":158648962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001768"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63980,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:39.724020+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:39.724020+0000","last_peered":"2026-03-31T21:24:39.724020+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:39.724020+0000","last_undegraded":"2026-03-31T21:24:39.724020+0000","last_fullsized":"2026-03-31T21:24:39.724020+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212209664,"data_stored":212209664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113228,"kb_used_data":47612,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258612,"statfs":{"total":96636764160,"available":96520818688,"internally_reserved":0,"allocated":48754688,"data_stored":48461014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47972352,"data_stored":47972352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:40.289 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:40.488 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:40.488 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:40.500 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":763,"stamp":"2026-03-31T21:24:39.973003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25201,"num_bytes_recovered":206703816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387384,"kb_used_data":155784,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728136,"statfs":{"total":289910292480,"available":289513611264,"internally_reserved":0,"allocated":159522816,"data_stored":158648962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001768"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63980,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:39.724020+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:39.724020+0000","last_peered":"2026-03-31T21:24:39.724020+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:39.724020+0000","last_undegraded":"2026-03-31T21:24:39.724020+0000","last_fullsized":"2026-03-31T21:24:39.724020+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212209664,"data_stored":212209664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267065,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113228,"kb_used_data":47612,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258612,"statfs":{"total":96636764160,"available":96520818688,"internally_reserved":0,"allocated":48754688,"data_stored":48461014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47972352,"data_stored":47972352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:43.501 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:43.692 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:43.692 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:43.704 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2026-03-31T21:24:41.973339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25201,"num_bytes_recovered":206703816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387384,"kb_used_data":155784,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728136,"statfs":{"total":289910292480,"available":289513611264,"internally_reserved":0,"allocated":159522816,"data_stored":158648962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001839"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63980,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:39.724020+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:39.724020+0000","last_peered":"2026-03-31T21:24:39.724020+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:39.724020+0000","last_undegraded":"2026-03-31T21:24:39.724020+0000","last_fullsized":"2026-03-31T21:24:39.724020+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212209664,"data_stored":212209664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113228,"kb_used_data":47612,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258612,"statfs":{"total":96636764160,"available":96520818688,"internally_reserved":0,"allocated":48754688,"data_stored":48461014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47972352,"data_stored":47972352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:43.704 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:43.892 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:43.892 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:43.903 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2026-03-31T21:24:41.973339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25201,"num_bytes_recovered":206703816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387384,"kb_used_data":155784,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282728136,"statfs":{"total":289910292480,"available":289513611264,"internally_reserved":0,"allocated":159522816,"data_stored":158648962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001839"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":63980,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:39.724020+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:39.724020+0000","last_peered":"2026-03-31T21:24:39.724020+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:39.724020+0000","last_undegraded":"2026-03-31T21:24:39.724020+0000","last_fullsized":"2026-03-31T21:24:39.724020+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1655,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25199,"num_bytes_recovered":206375624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212209664,"data_stored":212209664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757636,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113228,"kb_used_data":47612,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258612,"statfs":{"total":96636764160,"available":96520818688,"internally_reserved":0,"allocated":48754688,"data_stored":48461014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47972352,"data_stored":47972352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:46.904 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:47.098 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:47.098 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:47.110 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2026-03-31T21:24:45.973925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25250,"num_bytes_recovered":207105224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387572,"kb_used_data":155972,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727948,"statfs":{"total":289910292480,"available":289513418752,"internally_reserved":0,"allocated":159715328,"data_stored":158841474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001912"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64078,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:44.746850+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:44.746850+0000","last_peered":"2026-03-31T21:24:44.746850+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:44.746850+0000","last_undegraded":"2026-03-31T21:24:44.746850+0000","last_fullsized":"2026-03-31T21:24:44.746850+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25248,"num_bytes_recovered":206777032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25248,"num_bytes_recovered":206777032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212410368,"data_stored":212410368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313645,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757637,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113416,"kb_used_data":47800,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258424,"statfs":{"total":96636764160,"available":96520626176,"internally_reserved":0,"allocated":48947200,"data_stored":48653526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48173056,"data_stored":48173056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:47.110 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:47.307 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:47.307 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:47.319 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2026-03-31T21:24:45.973925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25250,"num_bytes_recovered":207105224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387572,"kb_used_data":155972,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727948,"statfs":{"total":289910292480,"available":289513418752,"internally_reserved":0,"allocated":159715328,"data_stored":158841474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001912"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64078,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:44.746850+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:44.746850+0000","last_peered":"2026-03-31T21:24:44.746850+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:44.746850+0000","last_undegraded":"2026-03-31T21:24:44.746850+0000","last_fullsized":"2026-03-31T21:24:44.746850+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25248,"num_bytes_recovered":206777032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1606,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25248,"num_bytes_recovered":206777032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212410368,"data_stored":212410368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430591,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267066,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313645,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757637,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113416,"kb_used_data":47800,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258424,"statfs":{"total":96636764160,"available":96520626176,"internally_reserved":0,"allocated":48947200,"data_stored":48653526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48173056,"data_stored":48173056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:50.320 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:50.597 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:50.597 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:50.609 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":768,"stamp":"2026-03-31T21:24:49.974410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25299,"num_bytes_recovered":207506632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387768,"kb_used_data":156168,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727752,"statfs":{"total":289910292480,"available":289513218048,"internally_reserved":0,"allocated":159916032,"data_stored":159042178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001749"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64176,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:49.751353+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:49.751353+0000","last_peered":"2026-03-31T21:24:49.751353+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:49.751353+0000","last_undegraded":"2026-03-31T21:24:49.751353+0000","last_fullsized":"2026-03-31T21:24:49.751353+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212611072,"data_stored":212611072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313646,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113612,"kb_used_data":47996,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258228,"statfs":{"total":96636764160,"available":96520425472,"internally_reserved":0,"allocated":49147904,"data_stored":48854230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48373760,"data_stored":48373760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:50.610 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:50.808 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:50.808 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:50.820 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":768,"stamp":"2026-03-31T21:24:49.974410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25299,"num_bytes_recovered":207506632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387768,"kb_used_data":156168,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727752,"statfs":{"total":289910292480,"available":289513218048,"internally_reserved":0,"allocated":159916032,"data_stored":159042178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001749"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64176,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:49.751353+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:49.751353+0000","last_peered":"2026-03-31T21:24:49.751353+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:49.751353+0000","last_undegraded":"2026-03-31T21:24:49.751353+0000","last_fullsized":"2026-03-31T21:24:49.751353+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212611072,"data_stored":212611072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267067,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313646,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113612,"kb_used_data":47996,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258228,"statfs":{"total":96636764160,"available":96520425472,"internally_reserved":0,"allocated":49147904,"data_stored":48854230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48373760,"data_stored":48373760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:53.821 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:54.013 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:54.013 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:54.024 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":770,"stamp":"2026-03-31T21:24:53.974881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25299,"num_bytes_recovered":207506632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387768,"kb_used_data":156168,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727752,"statfs":{"total":289910292480,"available":289513218048,"internally_reserved":0,"allocated":159916032,"data_stored":159042178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64176,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:49.751353+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:49.751353+0000","last_peered":"2026-03-31T21:24:49.751353+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:49.751353+0000","last_undegraded":"2026-03-31T21:24:49.751353+0000","last_fullsized":"2026-03-31T21:24:49.751353+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212611072,"data_stored":212611072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313647,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113612,"kb_used_data":47996,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258228,"statfs":{"total":96636764160,"available":96520425472,"internally_reserved":0,"allocated":49147904,"data_stored":48854230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48373760,"data_stored":48373760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:54.025 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:54.214 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:54.214 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:54.226 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":770,"stamp":"2026-03-31T21:24:53.974881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25299,"num_bytes_recovered":207506632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387768,"kb_used_data":156168,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727752,"statfs":{"total":289910292480,"available":289513218048,"internally_reserved":0,"allocated":159916032,"data_stored":159042178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001541"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64176,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:49.751353+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:49.751353+0000","last_peered":"2026-03-31T21:24:49.751353+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:49.751353+0000","last_undegraded":"2026-03-31T21:24:49.751353+0000","last_fullsized":"2026-03-31T21:24:49.751353+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1557,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25297,"num_bytes_recovered":207178440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212611072,"data_stored":212611072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430592,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313647,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757638,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113612,"kb_used_data":47996,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258228,"statfs":{"total":96636764160,"available":96520425472,"internally_reserved":0,"allocated":49147904,"data_stored":48854230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48373760,"data_stored":48373760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:57.227 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:57.423 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:57.423 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:57.436 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":771,"stamp":"2026-03-31T21:24:55.975161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1508,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25348,"num_bytes_recovered":207908040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387964,"kb_used_data":156364,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727556,"statfs":{"total":289910292480,"available":289513017344,"internally_reserved":0,"allocated":160116736,"data_stored":159242882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64274,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:54.760027+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:54.760027+0000","last_peered":"2026-03-31T21:24:54.760027+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:54.760027+0000","last_undegraded":"2026-03-31T21:24:54.760027+0000","last_fullsized":"2026-03-31T21:24:54.760027+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1508,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25346,"num_bytes_recovered":207579848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1508,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25346,"num_bytes_recovered":207579848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212807680,"data_stored":212807680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313647,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757639,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113808,"kb_used_data":48192,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258032,"statfs":{"total":96636764160,"available":96520224768,"internally_reserved":0,"allocated":49348608,"data_stored":49054934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48570368,"data_stored":48570368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:57.436 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:24:57.631 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:24:57.631 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:24:57.643 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":771,"stamp":"2026-03-31T21:24:55.975161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1508,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25348,"num_bytes_recovered":207908040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":387964,"kb_used_data":156364,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727556,"statfs":{"total":289910292480,"available":289513017344,"internally_reserved":0,"allocated":160116736,"data_stored":159242882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001509"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64274,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:54.760027+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:54.760027+0000","last_peered":"2026-03-31T21:24:54.760027+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:54.760027+0000","last_undegraded":"2026-03-31T21:24:54.760027+0000","last_fullsized":"2026-03-31T21:24:54.760027+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1508,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25346,"num_bytes_recovered":207579848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1508,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25346,"num_bytes_recovered":207579848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212807680,"data_stored":212807680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430593,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313647,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757639,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113808,"kb_used_data":48192,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94258032,"statfs":{"total":96636764160,"available":96520224768,"internally_reserved":0,"allocated":49348608,"data_stored":49054934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48570368,"data_stored":48570368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:24:59.823 INFO:tasks.ceph.mgr.x.vm00.stderr:2026-03-31T21:24:59.820+0000 7f8b7d694640 -1 calc_pg_upmaps abort due to max <= 0 2026-03-31T21:25:00.644 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:00.833 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:00.833 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:00.844 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":773,"stamp":"2026-03-31T21:24:59.975749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25397,"num_bytes_recovered":208309448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388152,"kb_used_data":156552,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727368,"statfs":{"total":289910292480,"available":289512824832,"internally_reserved":0,"allocated":160309248,"data_stored":159435394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64372,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:59.771501+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:59.771501+0000","last_peered":"2026-03-31T21:24:59.771501+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:59.771501+0000","last_undegraded":"2026-03-31T21:24:59.771501+0000","last_fullsized":"2026-03-31T21:24:59.771501+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213008384,"data_stored":213008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113996,"kb_used_data":48380,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257844,"statfs":{"total":96636764160,"available":96520032256,"internally_reserved":0,"allocated":49541120,"data_stored":49247446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48771072,"data_stored":48771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:00.845 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:01.036 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:01.036 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:01.048 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":773,"stamp":"2026-03-31T21:24:59.975749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25397,"num_bytes_recovered":208309448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388152,"kb_used_data":156552,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727368,"statfs":{"total":289910292480,"available":289512824832,"internally_reserved":0,"allocated":160309248,"data_stored":159435394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001571"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64372,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:59.771501+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:59.771501+0000","last_peered":"2026-03-31T21:24:59.771501+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:59.771501+0000","last_undegraded":"2026-03-31T21:24:59.771501+0000","last_fullsized":"2026-03-31T21:24:59.771501+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213008384,"data_stored":213008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267069,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113996,"kb_used_data":48380,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257844,"statfs":{"total":96636764160,"available":96520032256,"internally_reserved":0,"allocated":49541120,"data_stored":49247446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48771072,"data_stored":48771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:04.049 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:04.255 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:04.255 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:04.267 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":775,"stamp":"2026-03-31T21:25:03.976284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25397,"num_bytes_recovered":208309448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388152,"kb_used_data":156552,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727368,"statfs":{"total":289910292480,"available":289512824832,"internally_reserved":0,"allocated":160309248,"data_stored":159435394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001653"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64372,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:59.771501+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:59.771501+0000","last_peered":"2026-03-31T21:24:59.771501+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:59.771501+0000","last_undegraded":"2026-03-31T21:24:59.771501+0000","last_fullsized":"2026-03-31T21:24:59.771501+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213008384,"data_stored":213008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313649,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113996,"kb_used_data":48380,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257844,"statfs":{"total":96636764160,"available":96520032256,"internally_reserved":0,"allocated":49541120,"data_stored":49247446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48771072,"data_stored":48771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:04.267 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:04.470 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:04.470 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:04.483 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":775,"stamp":"2026-03-31T21:25:03.976284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25397,"num_bytes_recovered":208309448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388152,"kb_used_data":156552,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727368,"statfs":{"total":289910292480,"available":289512824832,"internally_reserved":0,"allocated":160309248,"data_stored":159435394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001653"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64372,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:24:59.771501+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:24:59.771501+0000","last_peered":"2026-03-31T21:24:59.771501+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:24:59.771501+0000","last_undegraded":"2026-03-31T21:24:59.771501+0000","last_fullsized":"2026-03-31T21:24:59.771501+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1459,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25395,"num_bytes_recovered":207981256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213008384,"data_stored":213008384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313649,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757640,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113996,"kb_used_data":48380,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257844,"statfs":{"total":96636764160,"available":96520032256,"internally_reserved":0,"allocated":49541120,"data_stored":49247446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48771072,"data_stored":48771072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:07.483 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:07.684 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:07.684 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:07.695 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2026-03-31T21:25:05.976536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1410,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25446,"num_bytes_recovered":208710856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388352,"kb_used_data":156752,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727168,"statfs":{"total":289910292480,"available":289512620032,"internally_reserved":0,"allocated":160514048,"data_stored":159640194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001654"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64470,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:04.776410+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:04.776410+0000","last_peered":"2026-03-31T21:25:04.776410+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:04.776410+0000","last_undegraded":"2026-03-31T21:25:04.776410+0000","last_fullsized":"2026-03-31T21:25:04.776410+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1410,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25444,"num_bytes_recovered":208382664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1410,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25444,"num_bytes_recovered":208382664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213209088,"data_stored":213209088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313649,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757641,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114196,"kb_used_data":48580,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257644,"statfs":{"total":96636764160,"available":96519827456,"internally_reserved":0,"allocated":49745920,"data_stored":49452246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48971776,"data_stored":48971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:07.696 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:07.893 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:07.893 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:07.905 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2026-03-31T21:25:05.976536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1410,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25446,"num_bytes_recovered":208710856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388352,"kb_used_data":156752,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282727168,"statfs":{"total":289910292480,"available":289512620032,"internally_reserved":0,"allocated":160514048,"data_stored":159640194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001654"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64470,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:04.776410+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:04.776410+0000","last_peered":"2026-03-31T21:25:04.776410+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:04.776410+0000","last_undegraded":"2026-03-31T21:25:04.776410+0000","last_fullsized":"2026-03-31T21:25:04.776410+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1410,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25444,"num_bytes_recovered":208382664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1410,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25444,"num_bytes_recovered":208382664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213209088,"data_stored":213209088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430596,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267070,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313649,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757641,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114196,"kb_used_data":48580,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257644,"statfs":{"total":96636764160,"available":96519827456,"internally_reserved":0,"allocated":49745920,"data_stored":49452246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48971776,"data_stored":48971776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:10.906 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:11.092 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:11.092 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:11.102 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":778,"stamp":"2026-03-31T21:25:09.977027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25494,"num_bytes_recovered":209104072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388544,"kb_used_data":156944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726976,"statfs":{"total":289910292480,"available":289512423424,"internally_reserved":0,"allocated":160710656,"data_stored":159836802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64566,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:09.803625+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:09.803625+0000","last_peered":"2026-03-31T21:25:09.803625+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:09.803625+0000","last_undegraded":"2026-03-31T21:25:09.803625+0000","last_fullsized":"2026-03-31T21:25:09.803625+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213405696,"data_stored":213405696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430597,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114388,"kb_used_data":48772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257452,"statfs":{"total":96636764160,"available":96519630848,"internally_reserved":0,"allocated":49942528,"data_stored":49648854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49168384,"data_stored":49168384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:11.102 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:11.283 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:11.283 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:11.292 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":778,"stamp":"2026-03-31T21:25:09.977027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25494,"num_bytes_recovered":209104072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388544,"kb_used_data":156944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726976,"statfs":{"total":289910292480,"available":289512423424,"internally_reserved":0,"allocated":160710656,"data_stored":159836802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001534"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64566,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:09.803625+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:09.803625+0000","last_peered":"2026-03-31T21:25:09.803625+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:09.803625+0000","last_undegraded":"2026-03-31T21:25:09.803625+0000","last_fullsized":"2026-03-31T21:25:09.803625+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213405696,"data_stored":213405696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430597,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267071,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114388,"kb_used_data":48772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257452,"statfs":{"total":96636764160,"available":96519630848,"internally_reserved":0,"allocated":49942528,"data_stored":49648854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49168384,"data_stored":49168384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:14.293 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:14.472 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:14.472 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:14.482 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":780,"stamp":"2026-03-31T21:25:13.977587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25494,"num_bytes_recovered":209104072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388544,"kb_used_data":156944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726976,"statfs":{"total":289910292480,"available":289512423424,"internally_reserved":0,"allocated":160710656,"data_stored":159836802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64566,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:09.803625+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:09.803625+0000","last_peered":"2026-03-31T21:25:09.803625+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:09.803625+0000","last_undegraded":"2026-03-31T21:25:09.803625+0000","last_fullsized":"2026-03-31T21:25:09.803625+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213405696,"data_stored":213405696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430597,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114388,"kb_used_data":48772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257452,"statfs":{"total":96636764160,"available":96519630848,"internally_reserved":0,"allocated":49942528,"data_stored":49648854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49168384,"data_stored":49168384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:14.482 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:14.665 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:14.665 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:14.675 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":780,"stamp":"2026-03-31T21:25:13.977587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25494,"num_bytes_recovered":209104072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388544,"kb_used_data":156944,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726976,"statfs":{"total":289910292480,"available":289512423424,"internally_reserved":0,"allocated":160710656,"data_stored":159836802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64566,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:09.803625+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:09.803625+0000","last_peered":"2026-03-31T21:25:09.803625+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:09.803625+0000","last_undegraded":"2026-03-31T21:25:09.803625+0000","last_fullsized":"2026-03-31T21:25:09.803625+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1362,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25492,"num_bytes_recovered":208775880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213405696,"data_stored":213405696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430597,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757642,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114388,"kb_used_data":48772,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257452,"statfs":{"total":96636764160,"available":96519630848,"internally_reserved":0,"allocated":49942528,"data_stored":49648854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49168384,"data_stored":49168384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:17.675 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:17.857 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:17.857 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:17.867 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":781,"stamp":"2026-03-31T21:25:15.977846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1315,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25541,"num_bytes_recovered":209489096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388740,"kb_used_data":157140,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726780,"statfs":{"total":289910292480,"available":289512222720,"internally_reserved":0,"allocated":160911360,"data_stored":160037506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001561"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64660,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:14.795913+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:14.795913+0000","last_peered":"2026-03-31T21:25:14.795913+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:14.795913+0000","last_undegraded":"2026-03-31T21:25:14.795913+0000","last_fullsized":"2026-03-31T21:25:14.795913+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25539,"num_bytes_recovered":209160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25539,"num_bytes_recovered":209160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213598208,"data_stored":213598208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430598,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757643,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114584,"kb_used_data":48968,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257256,"statfs":{"total":96636764160,"available":96519430144,"internally_reserved":0,"allocated":50143232,"data_stored":49849558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49360896,"data_stored":49360896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:17.867 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:18.047 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:18.047 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:18.058 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":782,"stamp":"2026-03-31T21:25:17.978084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1315,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25541,"num_bytes_recovered":209489096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388740,"kb_used_data":157140,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726780,"statfs":{"total":289910292480,"available":289512222720,"internally_reserved":0,"allocated":160911360,"data_stored":160037506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-95,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":778240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001547"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64660,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:14.795913+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:14.795913+0000","last_peered":"2026-03-31T21:25:14.795913+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:14.795913+0000","last_undegraded":"2026-03-31T21:25:14.795913+0000","last_fullsized":"2026-03-31T21:25:14.795913+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25539,"num_bytes_recovered":209160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1315,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25539,"num_bytes_recovered":209160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213598208,"data_stored":213598208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430598,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757643,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114584,"kb_used_data":48968,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257256,"statfs":{"total":96636764160,"available":96519430144,"internally_reserved":0,"allocated":50143232,"data_stored":49849558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49360896,"data_stored":49360896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:21.059 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:21.241 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:21.241 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:21.250 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2026-03-31T21:25:19.978334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25590,"num_bytes_recovered":209890504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388928,"kb_used_data":157328,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726592,"statfs":{"total":289910292480,"available":289512030208,"internally_reserved":0,"allocated":161103872,"data_stored":160230018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001547"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64758,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:19.779547+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:19.779547+0000","last_peered":"2026-03-31T21:25:19.779547+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:19.779547+0000","last_undegraded":"2026-03-31T21:25:19.779547+0000","last_fullsized":"2026-03-31T21:25:19.779547+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213798912,"data_stored":213798912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114772,"kb_used_data":49156,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257068,"statfs":{"total":96636764160,"available":96519237632,"internally_reserved":0,"allocated":50335744,"data_stored":50042070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49561600,"data_stored":49561600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:21.251 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:21.431 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:21.431 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:21.440 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2026-03-31T21:25:19.978334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25590,"num_bytes_recovered":209890504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388928,"kb_used_data":157328,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726592,"statfs":{"total":289910292480,"available":289512030208,"internally_reserved":0,"allocated":161103872,"data_stored":160230018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001547"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64758,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:19.779547+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:19.779547+0000","last_peered":"2026-03-31T21:25:19.779547+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:19.779547+0000","last_undegraded":"2026-03-31T21:25:19.779547+0000","last_fullsized":"2026-03-31T21:25:19.779547+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213798912,"data_stored":213798912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267073,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114772,"kb_used_data":49156,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257068,"statfs":{"total":96636764160,"available":96519237632,"internally_reserved":0,"allocated":50335744,"data_stored":50042070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49561600,"data_stored":49561600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:24.441 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:24.622 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:24.622 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:24.631 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":785,"stamp":"2026-03-31T21:25:23.978805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25590,"num_bytes_recovered":209890504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388928,"kb_used_data":157328,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726592,"statfs":{"total":289910292480,"available":289512030208,"internally_reserved":0,"allocated":161103872,"data_stored":160230018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64758,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:19.779547+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:19.779547+0000","last_peered":"2026-03-31T21:25:19.779547+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:19.779547+0000","last_undegraded":"2026-03-31T21:25:19.779547+0000","last_fullsized":"2026-03-31T21:25:19.779547+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213798912,"data_stored":213798912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313653,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114772,"kb_used_data":49156,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257068,"statfs":{"total":96636764160,"available":96519237632,"internally_reserved":0,"allocated":50335744,"data_stored":50042070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49561600,"data_stored":49561600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:24.631 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:24.810 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:24.810 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:24.820 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":785,"stamp":"2026-03-31T21:25:23.978805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25590,"num_bytes_recovered":209890504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":388928,"kb_used_data":157328,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726592,"statfs":{"total":289910292480,"available":289512030208,"internally_reserved":0,"allocated":161103872,"data_stored":160230018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001499"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64758,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:19.779547+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:19.779547+0000","last_peered":"2026-03-31T21:25:19.779547+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:19.779547+0000","last_undegraded":"2026-03-31T21:25:19.779547+0000","last_fullsized":"2026-03-31T21:25:19.779547+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1266,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25588,"num_bytes_recovered":209562312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213798912,"data_stored":213798912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430599,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267074,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313653,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757644,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114772,"kb_used_data":49156,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94257068,"statfs":{"total":96636764160,"available":96519237632,"internally_reserved":0,"allocated":50335744,"data_stored":50042070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49561600,"data_stored":49561600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:27.821 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:28.000 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:28.000 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:28.009 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":787,"stamp":"2026-03-31T21:25:27.979353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1217,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25639,"num_bytes_recovered":210291912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389124,"kb_used_data":157524,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726396,"statfs":{"total":289910292480,"available":289511829504,"internally_reserved":0,"allocated":161304576,"data_stored":160430722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001507"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64856,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:24.775386+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:24.775386+0000","last_peered":"2026-03-31T21:25:24.775386+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:24.775386+0000","last_undegraded":"2026-03-31T21:25:24.775386+0000","last_fullsized":"2026-03-31T21:25:24.775386+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25637,"num_bytes_recovered":209963720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25637,"num_bytes_recovered":209963720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213999616,"data_stored":213999616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430600,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757645,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114968,"kb_used_data":49352,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256872,"statfs":{"total":96636764160,"available":96519036928,"internally_reserved":0,"allocated":50536448,"data_stored":50242774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49762304,"data_stored":49762304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:28.009 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:28.188 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:28.188 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:28.198 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":787,"stamp":"2026-03-31T21:25:27.979353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1217,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25639,"num_bytes_recovered":210291912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389124,"kb_used_data":157524,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726396,"statfs":{"total":289910292480,"available":289511829504,"internally_reserved":0,"allocated":161304576,"data_stored":160430722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001507"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64856,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:24.775386+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:24.775386+0000","last_peered":"2026-03-31T21:25:24.775386+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:24.775386+0000","last_undegraded":"2026-03-31T21:25:24.775386+0000","last_fullsized":"2026-03-31T21:25:24.775386+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25637,"num_bytes_recovered":209963720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1217,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25637,"num_bytes_recovered":209963720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":213999616,"data_stored":213999616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430600,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757645,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114968,"kb_used_data":49352,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256872,"statfs":{"total":96636764160,"available":96519036928,"internally_reserved":0,"allocated":50536448,"data_stored":50242774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49762304,"data_stored":49762304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:31.198 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:31.385 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:31.385 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:31.396 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":788,"stamp":"2026-03-31T21:25:29.979607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25688,"num_bytes_recovered":210693320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389320,"kb_used_data":157720,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726200,"statfs":{"total":289910292480,"available":289511628800,"internally_reserved":0,"allocated":161505280,"data_stored":160631426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001523"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64954,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:29.761871+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:29.761871+0000","last_peered":"2026-03-31T21:25:29.761871+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:29.761871+0000","last_undegraded":"2026-03-31T21:25:29.761871+0000","last_fullsized":"2026-03-31T21:25:29.761871+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214200320,"data_stored":214200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430601,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757646,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115164,"kb_used_data":49548,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256676,"statfs":{"total":96636764160,"available":96518836224,"internally_reserved":0,"allocated":50737152,"data_stored":50443478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49963008,"data_stored":49963008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:31.397 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:31.575 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:31.575 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:31.585 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":788,"stamp":"2026-03-31T21:25:29.979607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25688,"num_bytes_recovered":210693320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389320,"kb_used_data":157720,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726200,"statfs":{"total":289910292480,"available":289511628800,"internally_reserved":0,"allocated":161505280,"data_stored":160631426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001523"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64954,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:29.761871+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:29.761871+0000","last_peered":"2026-03-31T21:25:29.761871+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:29.761871+0000","last_undegraded":"2026-03-31T21:25:29.761871+0000","last_fullsized":"2026-03-31T21:25:29.761871+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214200320,"data_stored":214200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430601,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267075,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757646,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115164,"kb_used_data":49548,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256676,"statfs":{"total":96636764160,"available":96518836224,"internally_reserved":0,"allocated":50737152,"data_stored":50443478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49963008,"data_stored":49963008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:34.586 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:34.765 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:34.765 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:34.775 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":790,"stamp":"2026-03-31T21:25:33.980129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25688,"num_bytes_recovered":210693320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389320,"kb_used_data":157720,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726200,"statfs":{"total":289910292480,"available":289511628800,"internally_reserved":0,"allocated":161505280,"data_stored":160631426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001573"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64954,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:29.761871+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:29.761871+0000","last_peered":"2026-03-31T21:25:29.761871+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:29.761871+0000","last_undegraded":"2026-03-31T21:25:29.761871+0000","last_fullsized":"2026-03-31T21:25:29.761871+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214200320,"data_stored":214200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430601,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313655,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757646,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115164,"kb_used_data":49548,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256676,"statfs":{"total":96636764160,"available":96518836224,"internally_reserved":0,"allocated":50737152,"data_stored":50443478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49963008,"data_stored":49963008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:34.775 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:34.954 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:34.954 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:34.963 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":790,"stamp":"2026-03-31T21:25:33.980129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25688,"num_bytes_recovered":210693320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389320,"kb_used_data":157720,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726200,"statfs":{"total":289910292480,"available":289511628800,"internally_reserved":0,"allocated":161505280,"data_stored":160631426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001573"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":64954,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:29.761871+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:29.761871+0000","last_peered":"2026-03-31T21:25:29.761871+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:29.761871+0000","last_undegraded":"2026-03-31T21:25:29.761871+0000","last_fullsized":"2026-03-31T21:25:29.761871+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":59,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:21:37.809403+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:21:37.809403+0000","last_peered":"2026-03-31T21:21:37.809403+0000","last_clean":"2026-03-31T21:21:37.809403+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:21:37.809403+0000","last_undegraded":"2026-03-31T21:21:37.809403+0000","last_fullsized":"2026-03-31T21:21:37.809403+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1168,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25686,"num_bytes_recovered":210365128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214200320,"data_stored":214200320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430601,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313655,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757646,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115164,"kb_used_data":49548,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256676,"statfs":{"total":96636764160,"available":96518836224,"internally_reserved":0,"allocated":50737152,"data_stored":50443478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49963008,"data_stored":49963008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:37.964 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:38.146 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:38.146 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:38.156 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":792,"stamp":"2026-03-31T21:25:37.980651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1119,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25737,"num_bytes_recovered":211094728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389516,"kb_used_data":157916,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726004,"statfs":{"total":289910292480,"available":289511428096,"internally_reserved":0,"allocated":161705984,"data_stored":160832130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65052,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:34.746660+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:34.746660+0000","last_peered":"2026-03-31T21:25:34.746660+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:34.746660+0000","last_undegraded":"2026-03-31T21:25:34.746660+0000","last_fullsized":"2026-03-31T21:25:34.746660+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25735,"num_bytes_recovered":210766536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25735,"num_bytes_recovered":210766536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214405120,"data_stored":214405120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430602,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757647,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115360,"kb_used_data":49744,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256480,"statfs":{"total":96636764160,"available":96518635520,"internally_reserved":0,"allocated":50937856,"data_stored":50644182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50167808,"data_stored":50167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:38.156 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:38.335 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:38.335 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:38.344 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":792,"stamp":"2026-03-31T21:25:37.980651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1119,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25737,"num_bytes_recovered":211094728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389516,"kb_used_data":157916,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282726004,"statfs":{"total":289910292480,"available":289511428096,"internally_reserved":0,"allocated":161705984,"data_stored":160832130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65052,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:34.746660+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:34.746660+0000","last_peered":"2026-03-31T21:25:34.746660+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:34.746660+0000","last_undegraded":"2026-03-31T21:25:34.746660+0000","last_fullsized":"2026-03-31T21:25:34.746660+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25735,"num_bytes_recovered":210766536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1119,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25735,"num_bytes_recovered":210766536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214405120,"data_stored":214405120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430602,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757647,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115360,"kb_used_data":49744,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256480,"statfs":{"total":96636764160,"available":96518635520,"internally_reserved":0,"allocated":50937856,"data_stored":50644182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50167808,"data_stored":50167808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:41.345 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:41.524 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:41.524 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:41.533 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":793,"stamp":"2026-03-31T21:25:39.980860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25786,"num_bytes_recovered":211496136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389712,"kb_used_data":158112,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725808,"statfs":{"total":289910292480,"available":289511227392,"internally_reserved":0,"allocated":161906688,"data_stored":161032834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65150,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:39.737034+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:39.737034+0000","last_peered":"2026-03-31T21:25:39.737034+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:39.737034+0000","last_undegraded":"2026-03-31T21:25:39.737034+0000","last_fullsized":"2026-03-31T21:25:39.737034+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214605824,"data_stored":214605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115556,"kb_used_data":49940,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256284,"statfs":{"total":96636764160,"available":96518434816,"internally_reserved":0,"allocated":51138560,"data_stored":50844886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50368512,"data_stored":50368512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:41.534 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:41.716 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:41.717 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:41.726 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":793,"stamp":"2026-03-31T21:25:39.980860+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25786,"num_bytes_recovered":211496136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389712,"kb_used_data":158112,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725808,"statfs":{"total":289910292480,"available":289511227392,"internally_reserved":0,"allocated":161906688,"data_stored":161032834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001506"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65150,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:39.737034+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:39.737034+0000","last_peered":"2026-03-31T21:25:39.737034+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:39.737034+0000","last_undegraded":"2026-03-31T21:25:39.737034+0000","last_fullsized":"2026-03-31T21:25:39.737034+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214605824,"data_stored":214605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267077,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115556,"kb_used_data":49940,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256284,"statfs":{"total":96636764160,"available":96518434816,"internally_reserved":0,"allocated":51138560,"data_stored":50844886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50368512,"data_stored":50368512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:44.728 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:44.908 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:44.908 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:44.917 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":795,"stamp":"2026-03-31T21:25:43.981325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25786,"num_bytes_recovered":211496136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389712,"kb_used_data":158112,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725808,"statfs":{"total":289910292480,"available":289511227392,"internally_reserved":0,"allocated":161906688,"data_stored":161032834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65150,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:39.737034+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:39.737034+0000","last_peered":"2026-03-31T21:25:39.737034+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:39.737034+0000","last_undegraded":"2026-03-31T21:25:39.737034+0000","last_fullsized":"2026-03-31T21:25:39.737034+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214605824,"data_stored":214605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313657,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115556,"kb_used_data":49940,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256284,"statfs":{"total":96636764160,"available":96518434816,"internally_reserved":0,"allocated":51138560,"data_stored":50844886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50368512,"data_stored":50368512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:44.917 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:45.100 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:45.100 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:45.110 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":795,"stamp":"2026-03-31T21:25:43.981325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25786,"num_bytes_recovered":211496136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389712,"kb_used_data":158112,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725808,"statfs":{"total":289910292480,"available":289511227392,"internally_reserved":0,"allocated":161906688,"data_stored":161032834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65150,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:39.737034+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:39.737034+0000","last_peered":"2026-03-31T21:25:39.737034+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:39.737034+0000","last_undegraded":"2026-03-31T21:25:39.737034+0000","last_fullsized":"2026-03-31T21:25:39.737034+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1070,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25784,"num_bytes_recovered":211167944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214605824,"data_stored":214605824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430603,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267078,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313657,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757648,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115556,"kb_used_data":49940,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256284,"statfs":{"total":96636764160,"available":96518434816,"internally_reserved":0,"allocated":51138560,"data_stored":50844886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50368512,"data_stored":50368512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:48.110 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:48.301 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:48.301 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:48.312 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":797,"stamp":"2026-03-31T21:25:47.981821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1021,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25835,"num_bytes_recovered":211897544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389908,"kb_used_data":158308,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725612,"statfs":{"total":289910292480,"available":289511026688,"internally_reserved":0,"allocated":162107392,"data_stored":161233538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65248,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:44.731163+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:44.731163+0000","last_peered":"2026-03-31T21:25:44.731163+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:44.731163+0000","last_undegraded":"2026-03-31T21:25:44.731163+0000","last_fullsized":"2026-03-31T21:25:44.731163+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25833,"num_bytes_recovered":211569352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25833,"num_bytes_recovered":211569352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214806528,"data_stored":214806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430604,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313658,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757649,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115752,"kb_used_data":50136,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256088,"statfs":{"total":96636764160,"available":96518234112,"internally_reserved":0,"allocated":51339264,"data_stored":51045590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50569216,"data_stored":50569216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:48.312 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:48.493 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:48.493 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:48.504 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":797,"stamp":"2026-03-31T21:25:47.981821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1021,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25835,"num_bytes_recovered":211897544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":389908,"kb_used_data":158308,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725612,"statfs":{"total":289910292480,"available":289511026688,"internally_reserved":0,"allocated":162107392,"data_stored":161233538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001426"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65248,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:44.731163+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:44.731163+0000","last_peered":"2026-03-31T21:25:44.731163+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:44.731163+0000","last_undegraded":"2026-03-31T21:25:44.731163+0000","last_fullsized":"2026-03-31T21:25:44.731163+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25833,"num_bytes_recovered":211569352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1021,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25833,"num_bytes_recovered":211569352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":214806528,"data_stored":214806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430604,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313658,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757649,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115752,"kb_used_data":50136,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94256088,"statfs":{"total":96636764160,"available":96518234112,"internally_reserved":0,"allocated":51339264,"data_stored":51045590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50569216,"data_stored":50569216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:51.505 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:51.683 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:51.683 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:51.692 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2026-03-31T21:25:49.982088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25884,"num_bytes_recovered":212298952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390104,"kb_used_data":158504,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725416,"statfs":{"total":289910292480,"available":289510825984,"internally_reserved":0,"allocated":162308096,"data_stored":161434242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65346,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:49.764473+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:49.764473+0000","last_peered":"2026-03-31T21:25:49.764473+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:49.764473+0000","last_undegraded":"2026-03-31T21:25:49.764473+0000","last_fullsized":"2026-03-31T21:25:49.764473+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215003136,"data_stored":215003136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313658,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115948,"kb_used_data":50332,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255892,"statfs":{"total":96636764160,"available":96518033408,"internally_reserved":0,"allocated":51539968,"data_stored":51246294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50765824,"data_stored":50765824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:51.693 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:51.877 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:51.877 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:51.887 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2026-03-31T21:25:49.982088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25884,"num_bytes_recovered":212298952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390104,"kb_used_data":158504,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725416,"statfs":{"total":289910292480,"available":289510825984,"internally_reserved":0,"allocated":162308096,"data_stored":161434242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001436"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65346,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:49.764473+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:49.764473+0000","last_peered":"2026-03-31T21:25:49.764473+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:49.764473+0000","last_undegraded":"2026-03-31T21:25:49.764473+0000","last_fullsized":"2026-03-31T21:25:49.764473+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215003136,"data_stored":215003136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267079,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313658,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115948,"kb_used_data":50332,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255892,"statfs":{"total":96636764160,"available":96518033408,"internally_reserved":0,"allocated":51539968,"data_stored":51246294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50765824,"data_stored":50765824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:54.888 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:55.068 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:55.068 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:55.078 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2026-03-31T21:25:53.982581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25884,"num_bytes_recovered":212298952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390104,"kb_used_data":158504,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725416,"statfs":{"total":289910292480,"available":289510825984,"internally_reserved":0,"allocated":162308096,"data_stored":161434242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001507"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65346,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:49.764473+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:49.764473+0000","last_peered":"2026-03-31T21:25:49.764473+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:49.764473+0000","last_undegraded":"2026-03-31T21:25:49.764473+0000","last_fullsized":"2026-03-31T21:25:49.764473+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215003136,"data_stored":215003136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313659,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115948,"kb_used_data":50332,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255892,"statfs":{"total":96636764160,"available":96518033408,"internally_reserved":0,"allocated":51539968,"data_stored":51246294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50765824,"data_stored":50765824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:55.078 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:55.345 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:25:55.342+0000 7fc677fff640 0 --2- 192.168.123.100:0/4080521846 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7fc67807f8c0 0x7fc67807fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:25:55.463 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:55.463 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:55.472 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2026-03-31T21:25:53.982581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25884,"num_bytes_recovered":212298952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390104,"kb_used_data":158504,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725416,"statfs":{"total":289910292480,"available":289510825984,"internally_reserved":0,"allocated":162308096,"data_stored":161434242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001507"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65346,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:49.764473+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:49.764473+0000","last_peered":"2026-03-31T21:25:49.764473+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:49.764473+0000","last_undegraded":"2026-03-31T21:25:49.764473+0000","last_fullsized":"2026-03-31T21:25:49.764473+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":972,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25882,"num_bytes_recovered":211970760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215003136,"data_stored":215003136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430605,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313659,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757650,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115948,"kb_used_data":50332,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255892,"statfs":{"total":96636764160,"available":96518033408,"internally_reserved":0,"allocated":51539968,"data_stored":51246294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50765824,"data_stored":50765824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:58.473 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:58.653 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:58.653 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:58.663 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":802,"stamp":"2026-03-31T21:25:57.983077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":923,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25933,"num_bytes_recovered":212700360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390300,"kb_used_data":158700,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725220,"statfs":{"total":289910292480,"available":289510625280,"internally_reserved":0,"allocated":162508800,"data_stored":161634946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65444,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:54.768484+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:54.768484+0000","last_peered":"2026-03-31T21:25:54.768484+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:54.768484+0000","last_undegraded":"2026-03-31T21:25:54.768484+0000","last_fullsized":"2026-03-31T21:25:54.768484+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":923,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25931,"num_bytes_recovered":212372168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":923,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25931,"num_bytes_recovered":212372168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215203840,"data_stored":215203840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430606,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267081,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313660,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116144,"kb_used_data":50528,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255696,"statfs":{"total":96636764160,"available":96517832704,"internally_reserved":0,"allocated":51740672,"data_stored":51446998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50966528,"data_stored":50966528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:58.663 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:25:58.840 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:25:58.840 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:25:58.851 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":802,"stamp":"2026-03-31T21:25:57.983077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":923,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25933,"num_bytes_recovered":212700360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390300,"kb_used_data":158700,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725220,"statfs":{"total":289910292480,"available":289510625280,"internally_reserved":0,"allocated":162508800,"data_stored":161634946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001513"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65444,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:54.768484+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:54.768484+0000","last_peered":"2026-03-31T21:25:54.768484+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:54.768484+0000","last_undegraded":"2026-03-31T21:25:54.768484+0000","last_fullsized":"2026-03-31T21:25:54.768484+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":923,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25931,"num_bytes_recovered":212372168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":923,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25931,"num_bytes_recovered":212372168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215203840,"data_stored":215203840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430606,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267081,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313660,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757651,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116144,"kb_used_data":50528,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255696,"statfs":{"total":96636764160,"available":96517832704,"internally_reserved":0,"allocated":51740672,"data_stored":51446998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50966528,"data_stored":50966528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:25:59.823 INFO:tasks.ceph.mgr.x.vm00.stderr:2026-03-31T21:25:59.821+0000 7f8b7d694640 -1 calc_pg_upmaps abort due to max <= 0 2026-03-31T21:26:01.852 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:02.035 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:02.035 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:02.044 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":804,"stamp":"2026-03-31T21:26:01.983567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25982,"num_bytes_recovered":213101768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390496,"kb_used_data":158896,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725024,"statfs":{"total":289910292480,"available":289510424576,"internally_reserved":0,"allocated":162709504,"data_stored":161835650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001478"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65543,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:59.756241+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:59.756241+0000","last_peered":"2026-03-31T21:25:59.756241+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:59.756241+0000","last_undegraded":"2026-03-31T21:25:59.756241+0000","last_fullsized":"2026-03-31T21:25:59.756241+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215408640,"data_stored":215408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430607,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267082,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313660,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116340,"kb_used_data":50724,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255500,"statfs":{"total":96636764160,"available":96517632000,"internally_reserved":0,"allocated":51941376,"data_stored":51647702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51171328,"data_stored":51171328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:02.044 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:02.227 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:02.228 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:02.237 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":804,"stamp":"2026-03-31T21:26:01.983567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25982,"num_bytes_recovered":213101768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390496,"kb_used_data":158896,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725024,"statfs":{"total":289910292480,"available":289510424576,"internally_reserved":0,"allocated":162709504,"data_stored":161835650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001478"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65543,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:59.756241+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:59.756241+0000","last_peered":"2026-03-31T21:25:59.756241+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:59.756241+0000","last_undegraded":"2026-03-31T21:25:59.756241+0000","last_fullsized":"2026-03-31T21:25:59.756241+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215408640,"data_stored":215408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430607,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267082,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313660,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116340,"kb_used_data":50724,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255500,"statfs":{"total":96636764160,"available":96517632000,"internally_reserved":0,"allocated":51941376,"data_stored":51647702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51171328,"data_stored":51171328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:05.238 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:05.423 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:05.423 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:05.433 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2026-03-31T21:26:03.983813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25982,"num_bytes_recovered":213101768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390496,"kb_used_data":158896,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725024,"statfs":{"total":289910292480,"available":289510424576,"internally_reserved":0,"allocated":162709504,"data_stored":161835650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65543,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:59.756241+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:59.756241+0000","last_peered":"2026-03-31T21:25:59.756241+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:59.756241+0000","last_undegraded":"2026-03-31T21:25:59.756241+0000","last_fullsized":"2026-03-31T21:25:59.756241+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215408640,"data_stored":215408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430607,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267082,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313661,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116340,"kb_used_data":50724,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255500,"statfs":{"total":96636764160,"available":96517632000,"internally_reserved":0,"allocated":51941376,"data_stored":51647702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51171328,"data_stored":51171328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:05.433 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:05.618 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:05.618 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:05.628 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2026-03-31T21:26:03.983813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25982,"num_bytes_recovered":213101768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390496,"kb_used_data":158896,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282725024,"statfs":{"total":289910292480,"available":289510424576,"internally_reserved":0,"allocated":162709504,"data_stored":161835650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65543,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:25:59.756241+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:25:59.756241+0000","last_peered":"2026-03-31T21:25:59.756241+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:25:59.756241+0000","last_undegraded":"2026-03-31T21:25:59.756241+0000","last_fullsized":"2026-03-31T21:25:59.756241+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25980,"num_bytes_recovered":212773576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215408640,"data_stored":215408640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430607,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267082,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313661,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757652,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116340,"kb_used_data":50724,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255500,"statfs":{"total":96636764160,"available":96517632000,"internally_reserved":0,"allocated":51941376,"data_stored":51647702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51171328,"data_stored":51171328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:08.628 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:08.811 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:08.811 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:08.822 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2026-03-31T21:26:07.984336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":825,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26031,"num_bytes_recovered":213503176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390696,"kb_used_data":159096,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724824,"statfs":{"total":289910292480,"available":289510219776,"internally_reserved":0,"allocated":162914304,"data_stored":162040450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001495"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65641,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:04.744953+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:04.744953+0000","last_peered":"2026-03-31T21:26:04.744953+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:04.744953+0000","last_undegraded":"2026-03-31T21:26:04.744953+0000","last_fullsized":"2026-03-31T21:26:04.744953+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":825,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26029,"num_bytes_recovered":213174984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":825,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26029,"num_bytes_recovered":213174984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215609344,"data_stored":215609344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430608,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267083,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313662,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757653,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116540,"kb_used_data":50924,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255300,"statfs":{"total":96636764160,"available":96517427200,"internally_reserved":0,"allocated":52146176,"data_stored":51852502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51372032,"data_stored":51372032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:08.823 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:09.009 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:09.009 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:09.020 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2026-03-31T21:26:07.984336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":825,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26031,"num_bytes_recovered":213503176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390696,"kb_used_data":159096,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724824,"statfs":{"total":289910292480,"available":289510219776,"internally_reserved":0,"allocated":162914304,"data_stored":162040450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001495"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65641,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:04.744953+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:04.744953+0000","last_peered":"2026-03-31T21:26:04.744953+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:04.744953+0000","last_undegraded":"2026-03-31T21:26:04.744953+0000","last_fullsized":"2026-03-31T21:26:04.744953+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":825,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26029,"num_bytes_recovered":213174984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":825,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26029,"num_bytes_recovered":213174984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215609344,"data_stored":215609344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430608,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267083,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313662,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757653,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116540,"kb_used_data":50924,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255300,"statfs":{"total":96636764160,"available":96517427200,"internally_reserved":0,"allocated":52146176,"data_stored":51852502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51372032,"data_stored":51372032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:12.021 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:12.206 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:12.206 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:12.216 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":809,"stamp":"2026-03-31T21:26:11.984774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":213904584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390892,"kb_used_data":159292,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724628,"statfs":{"total":289910292480,"available":289510019072,"internally_reserved":0,"allocated":163115008,"data_stored":162241154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001437"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65739,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:09.736013+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:09.736013+0000","last_peered":"2026-03-31T21:26:09.736013+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:09.736013+0000","last_undegraded":"2026-03-31T21:26:09.736013+0000","last_fullsized":"2026-03-31T21:26:09.736013+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215810048,"data_stored":215810048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313662,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116736,"kb_used_data":51120,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255104,"statfs":{"total":96636764160,"available":96517226496,"internally_reserved":0,"allocated":52346880,"data_stored":52053206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51572736,"data_stored":51572736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:12.216 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:12.399 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:12.399 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:12.409 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":809,"stamp":"2026-03-31T21:26:11.984774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":213904584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390892,"kb_used_data":159292,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724628,"statfs":{"total":289910292480,"available":289510019072,"internally_reserved":0,"allocated":163115008,"data_stored":162241154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001437"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65739,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:09.736013+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:09.736013+0000","last_peered":"2026-03-31T21:26:09.736013+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:09.736013+0000","last_undegraded":"2026-03-31T21:26:09.736013+0000","last_fullsized":"2026-03-31T21:26:09.736013+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215810048,"data_stored":215810048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313662,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116736,"kb_used_data":51120,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255104,"statfs":{"total":96636764160,"available":96517226496,"internally_reserved":0,"allocated":52346880,"data_stored":52053206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51572736,"data_stored":51572736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:15.410 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:15.591 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:15.591 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:15.602 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":810,"stamp":"2026-03-31T21:26:13.985016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":213904584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390892,"kb_used_data":159292,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724628,"statfs":{"total":289910292480,"available":289510019072,"internally_reserved":0,"allocated":163115008,"data_stored":162241154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65739,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:09.736013+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:09.736013+0000","last_peered":"2026-03-31T21:26:09.736013+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:09.736013+0000","last_undegraded":"2026-03-31T21:26:09.736013+0000","last_fullsized":"2026-03-31T21:26:09.736013+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215810048,"data_stored":215810048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313663,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116736,"kb_used_data":51120,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255104,"statfs":{"total":96636764160,"available":96517226496,"internally_reserved":0,"allocated":52346880,"data_stored":52053206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51572736,"data_stored":51572736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:15.603 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:15.788 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:15.788 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:15.799 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":810,"stamp":"2026-03-31T21:26:13.985016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":213904584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":390892,"kb_used_data":159292,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724628,"statfs":{"total":289910292480,"available":289510019072,"internally_reserved":0,"allocated":163115008,"data_stored":162241154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001449"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65739,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:09.736013+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:09.736013+0000","last_peered":"2026-03-31T21:26:09.736013+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:09.736013+0000","last_undegraded":"2026-03-31T21:26:09.736013+0000","last_fullsized":"2026-03-31T21:26:09.736013+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":776,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26078,"num_bytes_recovered":213576392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":215810048,"data_stored":215810048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430609,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267084,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313663,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116736,"kb_used_data":51120,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94255104,"statfs":{"total":96636764160,"available":96517226496,"internally_reserved":0,"allocated":52346880,"data_stored":52053206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51572736,"data_stored":51572736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:18.800 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:18.980 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:18.980 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:18.989 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":812,"stamp":"2026-03-31T21:26:17.985466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":727,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26129,"num_bytes_recovered":214305992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391088,"kb_used_data":159488,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724432,"statfs":{"total":289910292480,"available":289509818368,"internally_reserved":0,"allocated":163315712,"data_stored":162441858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001399"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65837,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:14.721872+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:14.721872+0000","last_peered":"2026-03-31T21:26:14.721872+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:14.721872+0000","last_undegraded":"2026-03-31T21:26:14.721872+0000","last_fullsized":"2026-03-31T21:26:14.721872+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":727,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26127,"num_bytes_recovered":213977800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":727,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26127,"num_bytes_recovered":213977800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216010752,"data_stored":216010752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430610,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267085,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757655,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116932,"kb_used_data":51316,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254908,"statfs":{"total":96636764160,"available":96517025792,"internally_reserved":0,"allocated":52547584,"data_stored":52253910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51773440,"data_stored":51773440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:18.989 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:19.175 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:19.175 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:19.185 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":812,"stamp":"2026-03-31T21:26:17.985466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":727,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26129,"num_bytes_recovered":214305992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391088,"kb_used_data":159488,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724432,"statfs":{"total":289910292480,"available":289509818368,"internally_reserved":0,"allocated":163315712,"data_stored":162441858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001399"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65837,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:14.721872+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:14.721872+0000","last_peered":"2026-03-31T21:26:14.721872+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:14.721872+0000","last_undegraded":"2026-03-31T21:26:14.721872+0000","last_fullsized":"2026-03-31T21:26:14.721872+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":727,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26127,"num_bytes_recovered":213977800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":727,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26127,"num_bytes_recovered":213977800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216010752,"data_stored":216010752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430610,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267085,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757655,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":116932,"kb_used_data":51316,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254908,"statfs":{"total":96636764160,"available":96517025792,"internally_reserved":0,"allocated":52547584,"data_stored":52253910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51773440,"data_stored":51773440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:22.186 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:22.370 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:22.370 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:22.380 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":814,"stamp":"2026-03-31T21:26:21.985961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26179,"num_bytes_recovered":214715592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391288,"kb_used_data":159688,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724232,"statfs":{"total":289910292480,"available":289509613568,"internally_reserved":0,"allocated":163520512,"data_stored":162646658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001394"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65937,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:19.817528+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:19.817528+0000","last_peered":"2026-03-31T21:26:19.817528+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:19.817528+0000","last_undegraded":"2026-03-31T21:26:19.817528+0000","last_fullsized":"2026-03-31T21:26:19.817528+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26177,"num_bytes_recovered":214387400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26177,"num_bytes_recovered":214387400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216211456,"data_stored":216211456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430612,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117132,"kb_used_data":51516,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254708,"statfs":{"total":96636764160,"available":96516820992,"internally_reserved":0,"allocated":52752384,"data_stored":52458710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51974144,"data_stored":51974144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:22.381 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:22.645 INFO:teuthology.orchestra.run.vm00.stderr:2026-03-31T21:26:22.643+0000 7fc11bfff640 0 --2- 192.168.123.100:0/860771473 >> [v2:192.168.123.100:3302/0,v1:192.168.123.100:6791/0] conn(0x7fc124080140 0x7fc124171fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2026-03-31T21:26:22.768 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:22.768 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:22.779 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":814,"stamp":"2026-03-31T21:26:21.985961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26179,"num_bytes_recovered":214715592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391288,"kb_used_data":159688,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724232,"statfs":{"total":289910292480,"available":289509613568,"internally_reserved":0,"allocated":163520512,"data_stored":162646658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001394"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65937,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:19.817528+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:19.817528+0000","last_peered":"2026-03-31T21:26:19.817528+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:19.817528+0000","last_undegraded":"2026-03-31T21:26:19.817528+0000","last_fullsized":"2026-03-31T21:26:19.817528+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26177,"num_bytes_recovered":214387400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26177,"num_bytes_recovered":214387400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216211456,"data_stored":216211456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430612,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117132,"kb_used_data":51516,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254708,"statfs":{"total":96636764160,"available":96516820992,"internally_reserved":0,"allocated":52752384,"data_stored":52458710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51974144,"data_stored":51974144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:25.780 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:25.969 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:25.969 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:25.979 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":815,"stamp":"2026-03-31T21:26:23.986185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26179,"num_bytes_recovered":214715592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391288,"kb_used_data":159688,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724232,"statfs":{"total":289910292480,"available":289509613568,"internally_reserved":0,"allocated":163520512,"data_stored":162646658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001410"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":65937,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:19.817528+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:19.817528+0000","last_peered":"2026-03-31T21:26:19.817528+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:19.817528+0000","last_undegraded":"2026-03-31T21:26:19.817528+0000","last_fullsized":"2026-03-31T21:26:19.817528+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26177,"num_bytes_recovered":214387400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":677,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26177,"num_bytes_recovered":214387400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216211456,"data_stored":216211456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430612,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313665,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757656,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117132,"kb_used_data":51516,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254708,"statfs":{"total":96636764160,"available":96516820992,"internally_reserved":0,"allocated":52752384,"data_stored":52458710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":51974144,"data_stored":51974144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:25.979 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:26.178 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:26.178 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:26.190 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2026-03-31T21:26:25.986436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26228,"num_bytes_recovered":215117000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391480,"kb_used_data":159880,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724040,"statfs":{"total":289910292480,"available":289509416960,"internally_reserved":0,"allocated":163717120,"data_stored":162843266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-148,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001419"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66035,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:24.804608+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:24.804608+0000","last_peered":"2026-03-31T21:26:24.804608+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:24.804608+0000","last_undegraded":"2026-03-31T21:26:24.804608+0000","last_fullsized":"2026-03-31T21:26:24.804608+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26226,"num_bytes_recovered":214788808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26226,"num_bytes_recovered":214788808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216412160,"data_stored":216412160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430613,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267086,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313665,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757657,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117324,"kb_used_data":51708,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254516,"statfs":{"total":96636764160,"available":96516624384,"internally_reserved":0,"allocated":52948992,"data_stored":52655318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52174848,"data_stored":52174848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:29.190 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:29.377 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:29.377 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:29.386 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":817,"stamp":"2026-03-31T21:26:27.986700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26228,"num_bytes_recovered":215117000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391480,"kb_used_data":159880,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724040,"statfs":{"total":289910292480,"available":289509416960,"internally_reserved":0,"allocated":163717120,"data_stored":162843266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66035,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:24.804608+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:24.804608+0000","last_peered":"2026-03-31T21:26:24.804608+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:24.804608+0000","last_undegraded":"2026-03-31T21:26:24.804608+0000","last_fullsized":"2026-03-31T21:26:24.804608+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26226,"num_bytes_recovered":214788808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26226,"num_bytes_recovered":214788808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216412160,"data_stored":216412160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430613,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267087,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757657,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117324,"kb_used_data":51708,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254516,"statfs":{"total":96636764160,"available":96516624384,"internally_reserved":0,"allocated":52948992,"data_stored":52655318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52174848,"data_stored":52174848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:29.386 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:29.565 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:29.566 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:29.575 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":817,"stamp":"2026-03-31T21:26:27.986700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26228,"num_bytes_recovered":215117000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391480,"kb_used_data":159880,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282724040,"statfs":{"total":289910292480,"available":289509416960,"internally_reserved":0,"allocated":163717120,"data_stored":162843266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-99,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":811008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001460"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66035,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:24.804608+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:24.804608+0000","last_peered":"2026-03-31T21:26:24.804608+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:24.804608+0000","last_undegraded":"2026-03-31T21:26:24.804608+0000","last_fullsized":"2026-03-31T21:26:24.804608+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26226,"num_bytes_recovered":214788808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":628,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26226,"num_bytes_recovered":214788808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216412160,"data_stored":216412160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430613,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267087,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757657,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117324,"kb_used_data":51708,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254516,"statfs":{"total":96636764160,"available":96516624384,"internally_reserved":0,"allocated":52948992,"data_stored":52655318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52174848,"data_stored":52174848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:32.576 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:32.772 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:32.772 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:32.784 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":819,"stamp":"2026-03-31T21:26:31.987216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":579,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26277,"num_bytes_recovered":215518408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391676,"kb_used_data":160076,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723844,"statfs":{"total":289910292480,"available":289509216256,"internally_reserved":0,"allocated":163917824,"data_stored":163043970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66133,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:29.798551+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:29.798551+0000","last_peered":"2026-03-31T21:26:29.798551+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:29.798551+0000","last_undegraded":"2026-03-31T21:26:29.798551+0000","last_fullsized":"2026-03-31T21:26:29.798551+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":579,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26275,"num_bytes_recovered":215190216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":579,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26275,"num_bytes_recovered":215190216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216612864,"data_stored":216612864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430614,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757658,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117520,"kb_used_data":51904,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254320,"statfs":{"total":96636764160,"available":96516423680,"internally_reserved":0,"allocated":53149696,"data_stored":52856022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52375552,"data_stored":52375552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:32.785 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:32.979 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:32.980 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:32.992 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":819,"stamp":"2026-03-31T21:26:31.987216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":579,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26277,"num_bytes_recovered":215518408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391676,"kb_used_data":160076,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723844,"statfs":{"total":289910292480,"available":289509216256,"internally_reserved":0,"allocated":163917824,"data_stored":163043970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66133,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:29.798551+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:29.798551+0000","last_peered":"2026-03-31T21:26:29.798551+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:29.798551+0000","last_undegraded":"2026-03-31T21:26:29.798551+0000","last_fullsized":"2026-03-31T21:26:29.798551+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":579,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26275,"num_bytes_recovered":215190216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":579,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26275,"num_bytes_recovered":215190216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216612864,"data_stored":216612864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430614,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757658,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117520,"kb_used_data":51904,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254320,"statfs":{"total":96636764160,"available":96516423680,"internally_reserved":0,"allocated":53149696,"data_stored":52856022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52375552,"data_stored":52375552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:35.993 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:36.177 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:36.177 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:36.187 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":821,"stamp":"2026-03-31T21:26:35.987731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26326,"num_bytes_recovered":215919816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391872,"kb_used_data":160272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723648,"statfs":{"total":289910292480,"available":289509015552,"internally_reserved":0,"allocated":164118528,"data_stored":163244674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001545"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66231,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:34.806496+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:34.806496+0000","last_peered":"2026-03-31T21:26:34.806496+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:34.806496+0000","last_undegraded":"2026-03-31T21:26:34.806496+0000","last_fullsized":"2026-03-31T21:26:34.806496+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216813568,"data_stored":216813568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430615,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313667,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757659,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117716,"kb_used_data":52100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254124,"statfs":{"total":96636764160,"available":96516222976,"internally_reserved":0,"allocated":53350400,"data_stored":53056726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52576256,"data_stored":52576256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:36.188 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:36.376 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:36.376 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:36.385 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":821,"stamp":"2026-03-31T21:26:35.987731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26326,"num_bytes_recovered":215919816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391872,"kb_used_data":160272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723648,"statfs":{"total":289910292480,"available":289509015552,"internally_reserved":0,"allocated":164118528,"data_stored":163244674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001545"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66231,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:34.806496+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:34.806496+0000","last_peered":"2026-03-31T21:26:34.806496+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:34.806496+0000","last_undegraded":"2026-03-31T21:26:34.806496+0000","last_fullsized":"2026-03-31T21:26:34.806496+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216813568,"data_stored":216813568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430615,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267088,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313667,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757659,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117716,"kb_used_data":52100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254124,"statfs":{"total":96636764160,"available":96516222976,"internally_reserved":0,"allocated":53350400,"data_stored":53056726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52576256,"data_stored":52576256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:39.386 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:39.571 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:39.571 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:39.582 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":822,"stamp":"2026-03-31T21:26:37.987993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26326,"num_bytes_recovered":215919816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391872,"kb_used_data":160272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723648,"statfs":{"total":289910292480,"available":289509015552,"internally_reserved":0,"allocated":164118528,"data_stored":163244674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66231,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:34.806496+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:34.806496+0000","last_peered":"2026-03-31T21:26:34.806496+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:34.806496+0000","last_undegraded":"2026-03-31T21:26:34.806496+0000","last_fullsized":"2026-03-31T21:26:34.806496+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216813568,"data_stored":216813568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430615,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267089,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757659,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117716,"kb_used_data":52100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254124,"statfs":{"total":96636764160,"available":96516222976,"internally_reserved":0,"allocated":53350400,"data_stored":53056726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52576256,"data_stored":52576256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:39.582 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:39.768 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:39.769 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:39.780 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":822,"stamp":"2026-03-31T21:26:37.987993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26326,"num_bytes_recovered":215919816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":391872,"kb_used_data":160272,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723648,"statfs":{"total":289910292480,"available":289509015552,"internally_reserved":0,"allocated":164118528,"data_stored":163244674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001556"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66231,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:34.806496+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:34.806496+0000","last_peered":"2026-03-31T21:26:34.806496+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:34.806496+0000","last_undegraded":"2026-03-31T21:26:34.806496+0000","last_fullsized":"2026-03-31T21:26:34.806496+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":530,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26324,"num_bytes_recovered":215591624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":216813568,"data_stored":216813568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430615,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267089,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757659,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117716,"kb_used_data":52100,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94254124,"statfs":{"total":96636764160,"available":96516222976,"internally_reserved":0,"allocated":53350400,"data_stored":53056726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52576256,"data_stored":52576256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:42.781 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:42.964 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:42.964 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:42.974 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":824,"stamp":"2026-03-31T21:26:41.988651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26374,"num_bytes_recovered":216313032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392068,"kb_used_data":160468,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723452,"statfs":{"total":289910292480,"available":289508814848,"internally_reserved":0,"allocated":164319232,"data_stored":163445378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001712"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66327,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:39.720371+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:39.720371+0000","last_peered":"2026-03-31T21:26:39.720371+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:39.720371+0000","last_undegraded":"2026-03-31T21:26:39.720371+0000","last_fullsized":"2026-03-31T21:26:39.720371+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26372,"num_bytes_recovered":215984840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26372,"num_bytes_recovered":215984840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217014272,"data_stored":217014272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430616,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757660,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117912,"kb_used_data":52296,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253928,"statfs":{"total":96636764160,"available":96516022272,"internally_reserved":0,"allocated":53551104,"data_stored":53257430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52776960,"data_stored":52776960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:42.974 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:43.154 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:43.154 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:43.163 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":824,"stamp":"2026-03-31T21:26:41.988651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26374,"num_bytes_recovered":216313032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392068,"kb_used_data":160468,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723452,"statfs":{"total":289910292480,"available":289508814848,"internally_reserved":0,"allocated":164319232,"data_stored":163445378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001712"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66327,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:39.720371+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:39.720371+0000","last_peered":"2026-03-31T21:26:39.720371+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:39.720371+0000","last_undegraded":"2026-03-31T21:26:39.720371+0000","last_fullsized":"2026-03-31T21:26:39.720371+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26372,"num_bytes_recovered":215984840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26372,"num_bytes_recovered":215984840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217014272,"data_stored":217014272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430616,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757660,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117912,"kb_used_data":52296,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253928,"statfs":{"total":96636764160,"available":96516022272,"internally_reserved":0,"allocated":53551104,"data_stored":53257430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52776960,"data_stored":52776960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:46.164 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:46.353 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:46.353 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:46.363 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":826,"stamp":"2026-03-31T21:26:45.989220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26423,"num_bytes_recovered":216714440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392260,"kb_used_data":160660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723260,"statfs":{"total":289910292480,"available":289508618240,"internally_reserved":0,"allocated":164515840,"data_stored":163641986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001761"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66425,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:44.771872+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:44.771872+0000","last_peered":"2026-03-31T21:26:44.771872+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:44.771872+0000","last_undegraded":"2026-03-31T21:26:44.771872+0000","last_fullsized":"2026-03-31T21:26:44.771872+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217210880,"data_stored":217210880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313669,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757661,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118104,"kb_used_data":52488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253736,"statfs":{"total":96636764160,"available":96515825664,"internally_reserved":0,"allocated":53747712,"data_stored":53454038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52973568,"data_stored":52973568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:46.363 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:46.549 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:46.550 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:46.561 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":826,"stamp":"2026-03-31T21:26:45.989220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26423,"num_bytes_recovered":216714440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392260,"kb_used_data":160660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723260,"statfs":{"total":289910292480,"available":289508618240,"internally_reserved":0,"allocated":164515840,"data_stored":163641986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-146,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001761"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66425,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:44.771872+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:44.771872+0000","last_peered":"2026-03-31T21:26:44.771872+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:44.771872+0000","last_undegraded":"2026-03-31T21:26:44.771872+0000","last_fullsized":"2026-03-31T21:26:44.771872+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217210880,"data_stored":217210880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267090,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313669,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757661,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118104,"kb_used_data":52488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253736,"statfs":{"total":96636764160,"available":96515825664,"internally_reserved":0,"allocated":53747712,"data_stored":53454038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52973568,"data_stored":52973568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:49.561 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:49.739 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:49.740 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:49.749 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":827,"stamp":"2026-03-31T21:26:47.989508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26423,"num_bytes_recovered":216714440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392260,"kb_used_data":160660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723260,"statfs":{"total":289910292480,"available":289508618240,"internally_reserved":0,"allocated":164515840,"data_stored":163641986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001777"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66425,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:44.771872+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:44.771872+0000","last_peered":"2026-03-31T21:26:44.771872+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:44.771872+0000","last_undegraded":"2026-03-31T21:26:44.771872+0000","last_fullsized":"2026-03-31T21:26:44.771872+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217210880,"data_stored":217210880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267091,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757661,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118104,"kb_used_data":52488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253736,"statfs":{"total":96636764160,"available":96515825664,"internally_reserved":0,"allocated":53747712,"data_stored":53454038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52973568,"data_stored":52973568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:49.749 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:49.928 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:49.928 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:49.938 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":827,"stamp":"2026-03-31T21:26:47.989508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26423,"num_bytes_recovered":216714440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392260,"kb_used_data":160660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723260,"statfs":{"total":289910292480,"available":289508618240,"internally_reserved":0,"allocated":164515840,"data_stored":163641986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001777"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66425,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:44.771872+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:44.771872+0000","last_peered":"2026-03-31T21:26:44.771872+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:44.771872+0000","last_undegraded":"2026-03-31T21:26:44.771872+0000","last_fullsized":"2026-03-31T21:26:44.771872+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":433,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26421,"num_bytes_recovered":216386248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217210880,"data_stored":217210880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430617,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267091,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757661,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118104,"kb_used_data":52488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253736,"statfs":{"total":96636764160,"available":96515825664,"internally_reserved":0,"allocated":53747712,"data_stored":53454038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52973568,"data_stored":52973568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:52.939 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:53.117 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:53.117 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:53.126 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2026-03-31T21:26:51.990043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":384,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26472,"num_bytes_recovered":217115848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392452,"kb_used_data":160852,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723068,"statfs":{"total":289910292480,"available":289508421632,"internally_reserved":0,"allocated":164712448,"data_stored":163838594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001707"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66523,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:49.763102+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:49.763102+0000","last_peered":"2026-03-31T21:26:49.763102+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:49.763102+0000","last_undegraded":"2026-03-31T21:26:49.763102+0000","last_fullsized":"2026-03-31T21:26:49.763102+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":384,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26470,"num_bytes_recovered":216787656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":384,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26470,"num_bytes_recovered":216787656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217415680,"data_stored":217415680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430618,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757662,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118296,"kb_used_data":52680,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253544,"statfs":{"total":96636764160,"available":96515629056,"internally_reserved":0,"allocated":53944320,"data_stored":53650646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53178368,"data_stored":53178368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:53.126 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:53.310 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:53.310 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:53.321 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2026-03-31T21:26:51.990043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":384,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26472,"num_bytes_recovered":217115848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392452,"kb_used_data":160852,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282723068,"statfs":{"total":289910292480,"available":289508421632,"internally_reserved":0,"allocated":164712448,"data_stored":163838594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001707"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66523,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:49.763102+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:49.763102+0000","last_peered":"2026-03-31T21:26:49.763102+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:49.763102+0000","last_undegraded":"2026-03-31T21:26:49.763102+0000","last_fullsized":"2026-03-31T21:26:49.763102+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":384,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26470,"num_bytes_recovered":216787656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":384,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26470,"num_bytes_recovered":216787656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217415680,"data_stored":217415680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430618,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757662,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118296,"kb_used_data":52680,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253544,"statfs":{"total":96636764160,"available":96515629056,"internally_reserved":0,"allocated":53944320,"data_stored":53650646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53178368,"data_stored":53178368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:56.322 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:56.516 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:56.516 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:56.528 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":831,"stamp":"2026-03-31T21:26:55.990589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26521,"num_bytes_recovered":217517256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392648,"kb_used_data":161048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722872,"statfs":{"total":289910292480,"available":289508220928,"internally_reserved":0,"allocated":164913152,"data_stored":164039298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001653"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66621,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:54.763953+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:54.763953+0000","last_peered":"2026-03-31T21:26:54.763953+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:54.763953+0000","last_undegraded":"2026-03-31T21:26:54.763953+0000","last_fullsized":"2026-03-31T21:26:54.763953+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26519,"num_bytes_recovered":217189064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26519,"num_bytes_recovered":217189064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217616384,"data_stored":217616384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430619,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757663,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118492,"kb_used_data":52876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253348,"statfs":{"total":96636764160,"available":96515428352,"internally_reserved":0,"allocated":54145024,"data_stored":53851350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":53379072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:56.528 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:56.720 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:56.720 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:56.732 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":831,"stamp":"2026-03-31T21:26:55.990589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26521,"num_bytes_recovered":217517256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392648,"kb_used_data":161048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722872,"statfs":{"total":289910292480,"available":289508220928,"internally_reserved":0,"allocated":164913152,"data_stored":164039298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-147,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001653"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66621,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:54.763953+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:54.763953+0000","last_peered":"2026-03-31T21:26:54.763953+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:54.763953+0000","last_undegraded":"2026-03-31T21:26:54.763953+0000","last_fullsized":"2026-03-31T21:26:54.763953+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26519,"num_bytes_recovered":217189064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26519,"num_bytes_recovered":217189064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217616384,"data_stored":217616384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430619,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757663,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118492,"kb_used_data":52876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253348,"statfs":{"total":96636764160,"available":96515428352,"internally_reserved":0,"allocated":54145024,"data_stored":53851350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":53379072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:59.733 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:26:59.824 INFO:tasks.ceph.mgr.x.vm00.stderr:2026-03-31T21:26:59.821+0000 7f8b7d694640 -1 calc_pg_upmaps abort due to max <= 0 2026-03-31T21:26:59.922 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:26:59.922 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:26:59.931 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":832,"stamp":"2026-03-31T21:26:57.990888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26521,"num_bytes_recovered":217517256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392648,"kb_used_data":161048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722872,"statfs":{"total":289910292480,"available":289508220928,"internally_reserved":0,"allocated":164913152,"data_stored":164039298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001667"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66621,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:54.763953+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:54.763953+0000","last_peered":"2026-03-31T21:26:54.763953+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:54.763953+0000","last_undegraded":"2026-03-31T21:26:54.763953+0000","last_fullsized":"2026-03-31T21:26:54.763953+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26519,"num_bytes_recovered":217189064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":335,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26519,"num_bytes_recovered":217189064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217616384,"data_stored":217616384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430619,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757663,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118492,"kb_used_data":52876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253348,"statfs":{"total":96636764160,"available":96515428352,"internally_reserved":0,"allocated":54145024,"data_stored":53851350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53379072,"data_stored":53379072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:26:59.932 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:00.121 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:00.121 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:00.131 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":833,"stamp":"2026-03-31T21:26:59.991145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26568,"num_bytes_recovered":217902280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392836,"kb_used_data":161236,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722684,"statfs":{"total":289910292480,"available":289508028416,"internally_reserved":0,"allocated":165105664,"data_stored":164231810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001636"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66715,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:59.756874+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:59.756874+0000","last_peered":"2026-03-31T21:26:59.756874+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:59.756874+0000","last_undegraded":"2026-03-31T21:26:59.756874+0000","last_fullsized":"2026-03-31T21:26:59.756874+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26566,"num_bytes_recovered":217574088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26566,"num_bytes_recovered":217574088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217808896,"data_stored":217808896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267093,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118680,"kb_used_data":53064,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253160,"statfs":{"total":96636764160,"available":96515235840,"internally_reserved":0,"allocated":54337536,"data_stored":54043862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53571584,"data_stored":53571584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:03.132 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:04.205 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:04.205 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:04.217 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":835,"stamp":"2026-03-31T21:27:03.991640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26568,"num_bytes_recovered":217902280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392836,"kb_used_data":161236,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722684,"statfs":{"total":289910292480,"available":289508028416,"internally_reserved":0,"allocated":165105664,"data_stored":164231810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001596"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66715,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:59.756874+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:59.756874+0000","last_peered":"2026-03-31T21:26:59.756874+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:59.756874+0000","last_undegraded":"2026-03-31T21:26:59.756874+0000","last_fullsized":"2026-03-31T21:26:59.756874+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26566,"num_bytes_recovered":217574088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26566,"num_bytes_recovered":217574088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217808896,"data_stored":217808896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313673,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118680,"kb_used_data":53064,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253160,"statfs":{"total":96636764160,"available":96515235840,"internally_reserved":0,"allocated":54337536,"data_stored":54043862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53571584,"data_stored":53571584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:04.217 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:04.414 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:04.414 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:04.426 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":835,"stamp":"2026-03-31T21:27:03.991640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26568,"num_bytes_recovered":217902280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":392836,"kb_used_data":161236,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722684,"statfs":{"total":289910292480,"available":289508028416,"internally_reserved":0,"allocated":165105664,"data_stored":164231810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-96,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001596"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66715,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:26:59.756874+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:26:59.756874+0000","last_peered":"2026-03-31T21:26:59.756874+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:26:59.756874+0000","last_undegraded":"2026-03-31T21:26:59.756874+0000","last_fullsized":"2026-03-31T21:26:59.756874+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26566,"num_bytes_recovered":217574088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":288,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26566,"num_bytes_recovered":217574088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217808896,"data_stored":217808896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430620,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313673,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757664,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118680,"kb_used_data":53064,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94253160,"statfs":{"total":96636764160,"available":96515235840,"internally_reserved":0,"allocated":54337536,"data_stored":54043862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53571584,"data_stored":53571584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:07.427 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:07.613 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:07.614 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:07.623 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":836,"stamp":"2026-03-31T21:27:05.991909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":240,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26616,"num_bytes_recovered":218295496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393064,"kb_used_data":161464,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722456,"statfs":{"total":289910292480,"available":289507794944,"internally_reserved":0,"allocated":165339136,"data_stored":164465282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66811,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:04.733409+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:04.733409+0000","last_peered":"2026-03-31T21:27:04.733409+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:04.733409+0000","last_undegraded":"2026-03-31T21:27:04.733409+0000","last_fullsized":"2026-03-31T21:27:04.733409+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":240,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26614,"num_bytes_recovered":217967304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":240,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26614,"num_bytes_recovered":217967304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218005504,"data_stored":218005504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430621,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313673,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757665,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118908,"kb_used_data":53292,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252932,"statfs":{"total":96636764160,"available":96515002368,"internally_reserved":0,"allocated":54571008,"data_stored":54277334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53768192,"data_stored":53768192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:07.623 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:07.811 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:07.811 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:07.822 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":836,"stamp":"2026-03-31T21:27:05.991909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":240,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26616,"num_bytes_recovered":218295496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393064,"kb_used_data":161464,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722456,"statfs":{"total":289910292480,"available":289507794944,"internally_reserved":0,"allocated":165339136,"data_stored":164465282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001589"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66811,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:04.733409+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:04.733409+0000","last_peered":"2026-03-31T21:27:04.733409+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:04.733409+0000","last_undegraded":"2026-03-31T21:27:04.733409+0000","last_fullsized":"2026-03-31T21:27:04.733409+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":240,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26614,"num_bytes_recovered":217967304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":240,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26614,"num_bytes_recovered":217967304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218005504,"data_stored":218005504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430621,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313673,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757665,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118908,"kb_used_data":53292,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252932,"statfs":{"total":96636764160,"available":96515002368,"internally_reserved":0,"allocated":54571008,"data_stored":54277334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53768192,"data_stored":53768192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:10.823 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:11.007 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:11.007 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:11.018 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":838,"stamp":"2026-03-31T21:27:09.992427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26665,"num_bytes_recovered":218696904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393260,"kb_used_data":161660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722260,"statfs":{"total":289910292480,"available":289507594240,"internally_reserved":0,"allocated":165539840,"data_stored":164665986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66909,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:09.734294+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:09.734294+0000","last_peered":"2026-03-31T21:27:09.734294+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:09.734294+0000","last_undegraded":"2026-03-31T21:27:09.734294+0000","last_fullsized":"2026-03-31T21:27:09.734294+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218206208,"data_stored":218206208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313674,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119104,"kb_used_data":53488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252736,"statfs":{"total":96636764160,"available":96514801664,"internally_reserved":0,"allocated":54771712,"data_stored":54478038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53968896,"data_stored":53968896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:11.019 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:11.200 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:11.200 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:11.211 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":838,"stamp":"2026-03-31T21:27:09.992427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26665,"num_bytes_recovered":218696904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393260,"kb_used_data":161660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722260,"statfs":{"total":289910292480,"available":289507594240,"internally_reserved":0,"allocated":165539840,"data_stored":164665986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-144,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66909,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:09.734294+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:09.734294+0000","last_peered":"2026-03-31T21:27:09.734294+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:09.734294+0000","last_undegraded":"2026-03-31T21:27:09.734294+0000","last_fullsized":"2026-03-31T21:27:09.734294+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218206208,"data_stored":218206208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267095,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313674,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119104,"kb_used_data":53488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252736,"statfs":{"total":96636764160,"available":96514801664,"internally_reserved":0,"allocated":54771712,"data_stored":54478038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53968896,"data_stored":53968896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:14.212 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:14.394 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:14.394 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:14.404 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":840,"stamp":"2026-03-31T21:27:13.992963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26665,"num_bytes_recovered":218696904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393260,"kb_used_data":161660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722260,"statfs":{"total":289910292480,"available":289507594240,"internally_reserved":0,"allocated":165539840,"data_stored":164665986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66909,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:09.734294+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:09.734294+0000","last_peered":"2026-03-31T21:27:09.734294+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:09.734294+0000","last_undegraded":"2026-03-31T21:27:09.734294+0000","last_fullsized":"2026-03-31T21:27:09.734294+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218206208,"data_stored":218206208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267096,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313675,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119104,"kb_used_data":53488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252736,"statfs":{"total":96636764160,"available":96514801664,"internally_reserved":0,"allocated":54771712,"data_stored":54478038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53968896,"data_stored":53968896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:14.404 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:14.588 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:14.588 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:14.599 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":840,"stamp":"2026-03-31T21:27:13.992963+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26665,"num_bytes_recovered":218696904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393260,"kb_used_data":161660,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722260,"statfs":{"total":289910292480,"available":289507594240,"internally_reserved":0,"allocated":165539840,"data_stored":164665986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":66909,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:09.734294+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:09.734294+0000","last_peered":"2026-03-31T21:27:09.734294+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:09.734294+0000","last_undegraded":"2026-03-31T21:27:09.734294+0000","last_fullsized":"2026-03-31T21:27:09.734294+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":191,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26663,"num_bytes_recovered":218368712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218206208,"data_stored":218206208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430622,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267096,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313675,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757666,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119104,"kb_used_data":53488,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252736,"statfs":{"total":96636764160,"available":96514801664,"internally_reserved":0,"allocated":54771712,"data_stored":54478038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53968896,"data_stored":53968896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:17.600 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:17.783 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:17.784 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:17.792 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":841,"stamp":"2026-03-31T21:27:15.993215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":143,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26713,"num_bytes_recovered":219090120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393452,"kb_used_data":161852,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722068,"statfs":{"total":289910292480,"available":289507397632,"internally_reserved":0,"allocated":165736448,"data_stored":164862594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001575"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67005,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:14.755361+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:14.755361+0000","last_peered":"2026-03-31T21:27:14.755361+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:14.755361+0000","last_undegraded":"2026-03-31T21:27:14.755361+0000","last_fullsized":"2026-03-31T21:27:14.755361+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26711,"num_bytes_recovered":218761928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26711,"num_bytes_recovered":218761928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218402816,"data_stored":218402816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430623,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267096,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313675,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757667,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119296,"kb_used_data":53680,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252544,"statfs":{"total":96636764160,"available":96514605056,"internally_reserved":0,"allocated":54968320,"data_stored":54674646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54165504,"data_stored":54165504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:17.793 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:18.861 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:18.861 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:18.872 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":842,"stamp":"2026-03-31T21:27:17.993500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":143,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26713,"num_bytes_recovered":219090120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393452,"kb_used_data":161852,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282722068,"statfs":{"total":289910292480,"available":289507397632,"internally_reserved":0,"allocated":165736448,"data_stored":164862594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001591"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67005,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:14.755361+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:14.755361+0000","last_peered":"2026-03-31T21:27:14.755361+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:14.755361+0000","last_undegraded":"2026-03-31T21:27:14.755361+0000","last_fullsized":"2026-03-31T21:27:14.755361+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26711,"num_bytes_recovered":218761928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":143,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26711,"num_bytes_recovered":218761928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218402816,"data_stored":218402816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430623,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267097,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313676,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757667,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119296,"kb_used_data":53680,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252544,"statfs":{"total":96636764160,"available":96514605056,"internally_reserved":0,"allocated":54968320,"data_stored":54674646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54165504,"data_stored":54165504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:21.873 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:22.063 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:22.063 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:22.074 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":844,"stamp":"2026-03-31T21:27:21.993985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26762,"num_bytes_recovered":219491528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393648,"kb_used_data":162048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282721872,"statfs":{"total":289910292480,"available":289507196928,"internally_reserved":0,"allocated":165937152,"data_stored":165063298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67103,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:19.774937+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:19.774937+0000","last_peered":"2026-03-31T21:27:19.774937+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:19.774937+0000","last_undegraded":"2026-03-31T21:27:19.774937+0000","last_fullsized":"2026-03-31T21:27:19.774937+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218599424,"data_stored":218599424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430624,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267098,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313676,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119492,"kb_used_data":53876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252348,"statfs":{"total":96636764160,"available":96514404352,"internally_reserved":0,"allocated":55169024,"data_stored":54875350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54362112,"data_stored":54362112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:22.074 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:22.265 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:22.265 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:22.276 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":844,"stamp":"2026-03-31T21:27:21.993985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26762,"num_bytes_recovered":219491528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393648,"kb_used_data":162048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282721872,"statfs":{"total":289910292480,"available":289507196928,"internally_reserved":0,"allocated":165937152,"data_stored":165063298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67103,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:19.774937+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:19.774937+0000","last_peered":"2026-03-31T21:27:19.774937+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:19.774937+0000","last_undegraded":"2026-03-31T21:27:19.774937+0000","last_fullsized":"2026-03-31T21:27:19.774937+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218599424,"data_stored":218599424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430624,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267098,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313676,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119492,"kb_used_data":53876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252348,"statfs":{"total":96636764160,"available":96514404352,"internally_reserved":0,"allocated":55169024,"data_stored":54875350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54362112,"data_stored":54362112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:25.277 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:25.460 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:25.460 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:25.470 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":845,"stamp":"2026-03-31T21:27:23.994307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26762,"num_bytes_recovered":219491528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393648,"kb_used_data":162048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282721872,"statfs":{"total":289910292480,"available":289507196928,"internally_reserved":0,"allocated":165937152,"data_stored":165063298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67103,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:19.774937+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:19.774937+0000","last_peered":"2026-03-31T21:27:19.774937+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:19.774937+0000","last_undegraded":"2026-03-31T21:27:19.774937+0000","last_fullsized":"2026-03-31T21:27:19.774937+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218599424,"data_stored":218599424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430624,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267098,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313677,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119492,"kb_used_data":53876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252348,"statfs":{"total":96636764160,"available":96514404352,"internally_reserved":0,"allocated":55169024,"data_stored":54875350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54362112,"data_stored":54362112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:25.470 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:25.656 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:25.656 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:25.666 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":845,"stamp":"2026-03-31T21:27:23.994307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26762,"num_bytes_recovered":219491528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393648,"kb_used_data":162048,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282721872,"statfs":{"total":289910292480,"available":289507196928,"internally_reserved":0,"allocated":165937152,"data_stored":165063298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-97,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":794624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001619"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67103,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:19.774937+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:19.774937+0000","last_peered":"2026-03-31T21:27:19.774937+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:19.774937+0000","last_undegraded":"2026-03-31T21:27:19.774937+0000","last_fullsized":"2026-03-31T21:27:19.774937+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":94,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26760,"num_bytes_recovered":219163336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218599424,"data_stored":218599424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430624,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267098,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313677,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119492,"kb_used_data":53876,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252348,"statfs":{"total":96636764160,"available":96514404352,"internally_reserved":0,"allocated":55169024,"data_stored":54875350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54362112,"data_stored":54362112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:28.667 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:28.847 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:28.847 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:28.857 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":847,"stamp":"2026-03-31T21:27:27.994779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26811,"num_bytes_recovered":219892936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393844,"kb_used_data":162244,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282721676,"statfs":{"total":289910292480,"available":289506996224,"internally_reserved":0,"allocated":166137856,"data_stored":165264002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001563"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67201,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:24.801014+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:24.801014+0000","last_peered":"2026-03-31T21:27:24.801014+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:24.801014+0000","last_undegraded":"2026-03-31T21:27:24.801014+0000","last_fullsized":"2026-03-31T21:27:24.801014+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26809,"num_bytes_recovered":219564744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26809,"num_bytes_recovered":219564744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218800128,"data_stored":218800128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430625,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313678,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757669,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119688,"kb_used_data":54072,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252152,"statfs":{"total":96636764160,"available":96514203648,"internally_reserved":0,"allocated":55369728,"data_stored":55076054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54562816,"data_stored":54562816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:28.857 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:29.035 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:29.035 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:29.046 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":847,"stamp":"2026-03-31T21:27:27.994779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26811,"num_bytes_recovered":219892936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":393844,"kb_used_data":162244,"kb_used_omap":58,"kb_used_meta":231493,"kb_avail":282721676,"statfs":{"total":289910292480,"available":289506996224,"internally_reserved":0,"allocated":166137856,"data_stored":165264002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237049329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-98,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001563"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67201,"reported_epoch":57,"state":"active+remapped+backfilling","last_fresh":"2026-03-31T21:27:24.801014+0000","last_change":"2026-03-31T21:20:04.108686+0000","last_active":"2026-03-31T21:27:24.801014+0000","last_peered":"2026-03-31T21:27:24.801014+0000","last_clean":"2026-03-31T21:04:08.969311+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:24.801014+0000","last_undegraded":"2026-03-31T21:27:24.801014+0000","last_fullsized":"2026-03-31T21:27:24.801014+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26809,"num_bytes_recovered":219564744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26809,"num_bytes_recovered":219564744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218800128,"data_stored":218800128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430625,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267099,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313678,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757669,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119688,"kb_used_data":54072,"kb_used_omap":17,"kb_used_meta":65582,"kb_avail":94252152,"statfs":{"total":96636764160,"available":96514203648,"internally_reserved":0,"allocated":55369728,"data_stored":55076054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67156300},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54562816,"data_stored":54562816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:32.047 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:32.283 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:32.283 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:32.305 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2026-03-31T21:27:31.995282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26855,"num_bytes_recovered":220253384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":394132,"kb_used_data":162404,"kb_used_omap":58,"kb_used_meta":231621,"kb_avail":282721388,"statfs":{"total":289910292480,"available":289506701312,"internally_reserved":0,"allocated":166301696,"data_stored":165427842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237180401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-94,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000974"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67293,"reported_epoch":57,"state":"active+clean+remapped","last_fresh":"2026-03-31T21:27:29.517988+0000","last_change":"2026-03-31T21:27:29.517765+0000","last_active":"2026-03-31T21:27:29.517988+0000","last_peered":"2026-03-31T21:27:29.517988+0000","last_clean":"2026-03-31T21:27:29.517988+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:29.517988+0000","last_undegraded":"2026-03-31T21:27:29.517988+0000","last_fullsized":"2026-03-31T21:27:29.517988+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218984448,"data_stored":218984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430626,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313678,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119976,"kb_used_data":54232,"kb_used_omap":17,"kb_used_meta":65710,"kb_avail":94251864,"statfs":{"total":96636764160,"available":96513908736,"internally_reserved":0,"allocated":55533568,"data_stored":55239894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67287372},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:32.305 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:recovered! 2026-03-31T21:27:32.305 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-03-31T21:27:32.307 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-03-31T21:27:32.307 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:32.522 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:32.522 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:32.533 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2026-03-31T21:27:31.995282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26855,"num_bytes_recovered":220253384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":394132,"kb_used_data":162404,"kb_used_omap":58,"kb_used_meta":231621,"kb_avail":282721388,"statfs":{"total":289910292480,"available":289506701312,"internally_reserved":0,"allocated":166301696,"data_stored":165427842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237180401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-94,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000974"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67293,"reported_epoch":57,"state":"active+clean+remapped","last_fresh":"2026-03-31T21:27:29.517988+0000","last_change":"2026-03-31T21:27:29.517765+0000","last_active":"2026-03-31T21:27:29.517988+0000","last_peered":"2026-03-31T21:27:29.517988+0000","last_clean":"2026-03-31T21:27:29.517988+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:29.517988+0000","last_undegraded":"2026-03-31T21:27:29.517988+0000","last_fullsized":"2026-03-31T21:27:29.517988+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218984448,"data_stored":218984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430626,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313678,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119976,"kb_used_data":54232,"kb_used_omap":17,"kb_used_meta":65710,"kb_avail":94251864,"statfs":{"total":96636764160,"available":96513908736,"internally_reserved":0,"allocated":55533568,"data_stored":55239894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67287372},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:32.533 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:32.734 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:32.734 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:32.745 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2026-03-31T21:27:31.995282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26855,"num_bytes_recovered":220253384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":394132,"kb_used_data":162404,"kb_used_omap":58,"kb_used_meta":231621,"kb_avail":282721388,"statfs":{"total":289910292480,"available":289506701312,"internally_reserved":0,"allocated":166301696,"data_stored":165427842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237180401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-94,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000974"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67293,"reported_epoch":57,"state":"active+clean+remapped","last_fresh":"2026-03-31T21:27:29.517988+0000","last_change":"2026-03-31T21:27:29.517765+0000","last_active":"2026-03-31T21:27:29.517988+0000","last_peered":"2026-03-31T21:27:29.517988+0000","last_clean":"2026-03-31T21:27:29.517988+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:29.517988+0000","last_undegraded":"2026-03-31T21:27:29.517988+0000","last_fullsized":"2026-03-31T21:27:29.517988+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218984448,"data_stored":218984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430626,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313678,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119976,"kb_used_data":54232,"kb_used_omap":17,"kb_used_meta":65710,"kb_avail":94251864,"statfs":{"total":96636764160,"available":96513908736,"internally_reserved":0,"allocated":55533568,"data_stored":55239894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67287372},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:32.745 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-03-31T21:27:32.745 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:32.932 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:32.932 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:32.943 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":851,"stamp":"2026-03-31T21:27:31.995282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26855,"num_bytes_recovered":220253384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":394132,"kb_used_data":162404,"kb_used_omap":58,"kb_used_meta":231621,"kb_avail":282721388,"statfs":{"total":289910292480,"available":289506701312,"internally_reserved":0,"allocated":166301696,"data_stored":165427842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59919,"internal_metadata":237180401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-94,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000974"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67293,"reported_epoch":57,"state":"active+clean+remapped","last_fresh":"2026-03-31T21:27:29.517988+0000","last_change":"2026-03-31T21:27:29.517765+0000","last_active":"2026-03-31T21:27:29.517988+0000","last_peered":"2026-03-31T21:27:29.517988+0000","last_clean":"2026-03-31T21:27:29.517988+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:29.517988+0000","last_undegraded":"2026-03-31T21:27:29.517988+0000","last_fullsized":"2026-03-31T21:27:29.517988+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":60,"reported_epoch":57,"state":"active+clean","last_fresh":"2026-03-31T21:23:37.817503+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:23:37.817503+0000","last_peered":"2026-03-31T21:23:37.817503+0000","last_clean":"2026-03-31T21:23:37.817503+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:23:37.817503+0000","last_undegraded":"2026-03-31T21:23:37.817503+0000","last_fullsized":"2026-03-31T21:23:37.817503+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218984448,"data_stored":218984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430626,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313678,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189940,"kb_used_data":54244,"kb_used_omap":27,"kb_used_meta":135652,"kb_avail":94181900,"statfs":{"total":96636764160,"available":96442265600,"internally_reserved":0,"allocated":55545856,"data_stored":55255766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28527,"internal_metadata":138907793},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119976,"kb_used_data":54232,"kb_used_omap":17,"kb_used_meta":65710,"kb_avail":94251864,"statfs":{"total":96636764160,"available":96513908736,"internally_reserved":0,"allocated":55533568,"data_stored":55239894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67287372},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:32.943 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-03-31T21:27:33.128 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:33.128 INFO:teuthology.orchestra.run.vm00.stdout:{"epoch":59,"fsid":"edf0be71-aa1c-40ef-819b-fbd038c1604f","created":"2026-03-31T21:00:56.876676+0000","modified":"2026-03-31T21:27:31.206593+0000","last_up_change":"2026-03-31T21:04:07.667010+0000","last_in_change":"2026-03-31T21:04:11.978235+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-03-31T21:01:03.806955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-03-31T21:01:11.337140+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"403a5788-99d7-42ee-96ed-c2891fdebaf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":34,"up_from":38,"up_thru":38,"down_at":35,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6812","nonce":681364674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6813","nonce":681364674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6815","nonce":681364674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6814","nonce":681364674}]},"public_addr":"192.168.123.100:6812/681364674","cluster_addr":"192.168.123.100:6813/681364674","heartbeat_back_addr":"192.168.123.100:6815/681364674","heartbeat_front_addr":"192.168.123.100:6814/681364674","state":["exists","up"]},{"osd":1,"uuid":"c1dccf3e-c41f-46a2-8fbd-09a0651739c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":18,"up_from":21,"up_thru":45,"down_at":19,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6800","nonce":2701095802}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6801","nonce":2701095802}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6803","nonce":2701095802}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6802","nonce":2701095802}]},"public_addr":"192.168.123.100:6800/2701095802","cluster_addr":"192.168.123.100:6801/2701095802","heartbeat_back_addr":"192.168.123.100:6803/2701095802","heartbeat_front_addr":"192.168.123.100:6802/2701095802","state":["exists","up"]},{"osd":2,"uuid":"f9e6dead-91ad-448b-b46d-36fdf906cfa7","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":49,"up_from":53,"up_thru":35,"down_at":50,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6808","nonce":2560261943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6809","nonce":2560261943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6811","nonce":2560261943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6810","nonce":2560261943}]},"public_addr":"192.168.123.100:6808/2560261943","cluster_addr":"192.168.123.100:6809/2560261943","heartbeat_back_addr":"192.168.123.100:6811/2560261943","heartbeat_front_addr":"192.168.123.100:6810/2560261943","state":["exists","up"]},{"osd":3,"uuid":"3b7f3900-5638-4954-97cf-d86d7e1e2462","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":44,"up_from":48,"up_thru":58,"down_at":45,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6804","nonce":3199358350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6805","nonce":3199358350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6807","nonce":3199358350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"192.168.123.100:6806","nonce":3199358350}]},"public_addr":"192.168.123.100:6804/3199358350","cluster_addr":"192.168.123.100:6805/3199358350","heartbeat_back_addr":"192.168.123.100:6807/3199358350","heartbeat_front_addr":"192.168.123.100:6806/3199358350","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.978308+0000","dead_epoch":35},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.991163+0000","dead_epoch":19},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":65536,"last_purged_snaps_scrub":"2026-03-31T21:00:59.932123+0000","dead_epoch":50},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4541880224203014143,"old_weight":0,"last_purged_snaps_scrub":"2026-03-31T21:00:59.868703+0000","dead_epoch":45}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-03-31T21:27:34.138 INFO:tasks.ceph:Scrubbing osd.0 2026-03-31T21:27:34.138 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-03-31T21:27:34.214 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:27:34.214 INFO:teuthology.orchestra.run.vm00.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-03-31T21:27:34.214 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:27:34.221 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-03-31T21:27:34.405 INFO:teuthology.orchestra.run.vm00.stderr:instructed osd(s) 0 to deep-scrub 2026-03-31T21:27:34.414 INFO:tasks.ceph:Scrubbing osd.1 2026-03-31T21:27:34.414 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-03-31T21:27:34.495 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:27:34.495 INFO:teuthology.orchestra.run.vm00.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-03-31T21:27:34.495 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:27:34.502 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-03-31T21:27:34.687 INFO:teuthology.orchestra.run.vm00.stderr:instructed osd(s) 1 to deep-scrub 2026-03-31T21:27:34.698 INFO:tasks.ceph:Scrubbing osd.2 2026-03-31T21:27:34.698 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-03-31T21:27:34.779 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:27:34.779 INFO:teuthology.orchestra.run.vm00.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-03-31T21:27:34.779 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:27:34.786 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-03-31T21:27:34.976 INFO:teuthology.orchestra.run.vm00.stderr:instructed osd(s) 2 to deep-scrub 2026-03-31T21:27:34.987 INFO:tasks.ceph:Scrubbing osd.3 2026-03-31T21:27:34.987 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2026-03-31T21:27:35.066 INFO:teuthology.orchestra.run.vm00.stdout:{ 2026-03-31T21:27:35.066 INFO:teuthology.orchestra.run.vm00.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-03-31T21:27:35.066 INFO:teuthology.orchestra.run.vm00.stdout:} 2026-03-31T21:27:35.073 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2026-03-31T21:27:35.255 INFO:teuthology.orchestra.run.vm00.stderr:instructed osd(s) 3 to deep-scrub 2026-03-31T21:27:35.265 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:35.446 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:35.446 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:35.456 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":852,"stamp":"2026-03-31T21:27:33.995549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109807460,"num_objects":13369,"num_object_clones":0,"num_object_copies":40105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13369,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":13404,"num_write_kb":107237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26855,"num_bytes_recovered":220253384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44,"ondisk_log_size":44,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":394164,"kb_used_data":162420,"kb_used_omap":59,"kb_used_meta":231620,"kb_avail":282721356,"statfs":{"total":289910292480,"available":289506668544,"internally_reserved":0,"allocated":166318080,"data_stored":165434615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61223,"internal_metadata":237179097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-45,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":360448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000994"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":67293,"reported_epoch":57,"state":"active+clean+remapped","last_fresh":"2026-03-31T21:27:29.517988+0000","last_change":"2026-03-31T21:27:29.517765+0000","last_active":"2026-03-31T21:27:29.517988+0000","last_peered":"2026-03-31T21:27:29.517988+0000","last_clean":"2026-03-31T21:27:29.517988+0000","last_became_active":"2026-03-31T21:04:14.041366+0000","last_became_peered":"2026-03-31T21:04:14.041366+0000","last_unstale":"2026-03-31T21:27:29.517988+0000","last_undegraded":"2026-03-31T21:27:29.517988+0000","last_fullsized":"2026-03-31T21:27:29.517988+0000","mapping_epoch":56,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"18'2","last_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_deep_scrub":"18'2","last_deep_scrub_stamp":"2026-03-31T21:01:39.870036+0000","last_clean_scrub_stamp":"2026-03-31T21:01:39.870036+0000","objects_scrubbed":2,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","0(2)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":13367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"11'17","reported_seq":64,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-03-31T21:27:31.213022+0000","last_change":"2026-03-31T21:03:36.186578+0000","last_active":"2026-03-31T21:27:31.213022+0000","last_peered":"2026-03-31T21:27:31.213022+0000","last_clean":"2026-03-31T21:27:31.213022+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:27:31.213022+0000","last_undegraded":"2026-03-31T21:27:31.213022+0000","last_fullsized":"2026-03-31T21:27:31.213022+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-03-31T21:01:04.289853+0000","last_clean_scrub_stamp":"2026-03-31T21:01:04.289853+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T03:21:48.306954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109479268,"num_objects":13367,"num_object_clones":0,"num_object_copies":40101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13367,"num_write_kb":106915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26853,"num_bytes_recovered":219925192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218984448,"data_stored":218984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430626,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84216,"kb_used_data":53928,"kb_used_omap":12,"kb_used_meta":30259,"kb_avail":94287624,"statfs":{"total":96636764160,"available":96550526976,"internally_reserved":0,"allocated":55222272,"data_stored":54932182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13292,"internal_metadata":30985236},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267100,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313679,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189972,"kb_used_data":54260,"kb_used_omap":29,"kb_used_meta":135650,"kb_avail":94181868,"statfs":{"total":96636764160,"available":96442232832,"internally_reserved":0,"allocated":55562240,"data_stored":55262539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29831,"internal_metadata":138906489},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":119976,"kb_used_data":54232,"kb_used_omap":17,"kb_used_meta":65710,"kb_avail":94251864,"statfs":{"total":96636764160,"available":96513908736,"internally_reserved":0,"allocated":55533568,"data_stored":55239894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18100,"internal_metadata":67287372},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1715,"internal_metadata":0}]}} 2026-03-31T21:27:35.457 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-03-31T21:01:39.870036+0000 time.struct_time(tm_year=2026, tm_mon=3, tm_mday=31, tm_hour=21, tm_min=1, tm_sec=39, tm_wday=1, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=3, tm_mday=31, tm_hour=21, tm_min=27, tm_sec=33, tm_wday=1, tm_yday=90, tm_isdst=0) 2026-03-31T21:27:35.457 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-03-31T21:01:04.289853+0000 time.struct_time(tm_year=2026, tm_mon=3, tm_mday=31, tm_hour=21, tm_min=1, tm_sec=4, tm_wday=1, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=3, tm_mday=31, tm_hour=21, tm_min=27, tm_sec=33, tm_wday=1, tm_yday=90, tm_isdst=0) 2026-03-31T21:27:35.458 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-03-31T21:27:55.458 DEBUG:teuthology.orchestra.run.vm00:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-03-31T21:27:55.904 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:27:55.904 INFO:teuthology.orchestra.run.vm00.stderr:dumped all 2026-03-31T21:27:55.914 INFO:teuthology.orchestra.run.vm00.stdout:{"pg_ready":true,"pg_map":{"version":862,"stamp":"2026-03-31T21:27:53.998219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109806782,"num_objects":13368,"num_object_clones":0,"num_object_copies":40102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13368,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":394264,"kb_used_data":162472,"kb_used_omap":62,"kb_used_meta":231681,"kb_avail":282721256,"statfs":{"total":289910292480,"available":289506566144,"internally_reserved":0,"allocated":166371328,"data_stored":165468641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63940,"internal_metadata":237241916},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001649"},"pg_stats":[{"pgid":"2.0","version":"57'13368","reported_seq":68344,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-03-31T21:27:38.693245+0000","last_change":"2026-03-31T21:27:38.693245+0000","last_active":"2026-03-31T21:27:38.693245+0000","last_peered":"2026-03-31T21:27:38.693245+0000","last_clean":"2026-03-31T21:27:38.693245+0000","last_became_active":"2026-03-31T21:27:31.213860+0000","last_became_peered":"2026-03-31T21:27:31.213860+0000","last_unstale":"2026-03-31T21:27:38.693245+0000","last_undegraded":"2026-03-31T21:27:38.693245+0000","last_fullsized":"2026-03-31T21:27:38.693245+0000","mapping_epoch":58,"log_start":"22'13341","ondisk_log_start":"22'13341","created":15,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'13368","last_scrub_stamp":"2026-03-31T21:27:38.693175+0000","last_deep_scrub":"57'13368","last_deep_scrub_stamp":"2026-03-31T21:27:38.693175+0000","last_clean_scrub_stamp":"2026-03-31T21:27:38.693175+0000","objects_scrubbed":13366,"log_size":27,"log_dups_size":3000,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2026-04-01T21:36:35.787446+0000","scrub_duration":2865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109478590,"num_objects":13366,"num_object_clones":0,"num_object_copies":40098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"59'19","reported_seq":75,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-03-31T21:27:34.930675+0000","last_change":"2026-03-31T21:27:34.930636+0000","last_active":"2026-03-31T21:27:34.930675+0000","last_peered":"2026-03-31T21:27:34.930675+0000","last_clean":"2026-03-31T21:27:34.930675+0000","last_became_active":"2026-03-31T21:03:35.880296+0000","last_became_peered":"2026-03-31T21:03:35.880296+0000","last_unstale":"2026-03-31T21:27:34.930675+0000","last_undegraded":"2026-03-31T21:27:34.930675+0000","last_fullsized":"2026-03-31T21:27:34.930675+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'19","last_scrub_stamp":"2026-03-31T21:27:34.930601+0000","last_deep_scrub":"59'19","last_deep_scrub_stamp":"2026-03-31T21:27:34.930601+0000","last_clean_scrub_stamp":"2026-03-31T21:27:34.930601+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-04-02T08:20:30.672664+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":109478590,"num_objects":13366,"num_object_clones":0,"num_object_copies":40098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218984448,"data_stored":218984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5250,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":991232,"data_stored":984064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":48,"seq":206158430630,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84232,"kb_used_data":53944,"kb_used_omap":14,"kb_used_meta":30257,"kb_avail":94287608,"statfs":{"total":96636764160,"available":96550510592,"internally_reserved":0,"allocated":55238656,"data_stored":54938955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14705,"internal_metadata":30983823},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":227633267104,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":21,"seq":90194313683,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189956,"kb_used_data":54260,"kb_used_omap":29,"kb_used_meta":135650,"kb_avail":94181884,"statfs":{"total":96636764160,"available":96442249216,"internally_reserved":0,"allocated":55562240,"data_stored":55262539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":29831,"internal_metadata":138906489},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":38,"seq":163208757674,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120076,"kb_used_data":54268,"kb_used_omap":18,"kb_used_meta":65773,"kb_avail":94251764,"statfs":{"total":96636764160,"available":96513806336,"internally_reserved":0,"allocated":55570432,"data_stored":55267147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19404,"internal_metadata":67351604},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331776,"data_stored":328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":54743040,"data_stored":54743040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54747136,"data_stored":54747136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5250,"internal_metadata":0}]}} 2026-03-31T21:27:55.915 DEBUG:teuthology.orchestra.run.vm00:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-03-31T21:27:56.124 INFO:teuthology.misc:Shutting down mds daemons... 2026-03-31T21:27:56.124 INFO:teuthology.misc:Shutting down osd daemons... 2026-03-31T21:27:56.124 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-03-31T21:27:56.125 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.156 INFO:tasks.ceph.osd.0:Stopped 2026-03-31T21:27:56.156 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-03-31T21:27:56.156 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.230 INFO:tasks.ceph.osd.1:Stopped 2026-03-31T21:27:56.230 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-03-31T21:27:56.230 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.306 INFO:tasks.ceph.osd.2:Stopped 2026-03-31T21:27:56.306 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-03-31T21:27:56.306 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.394 INFO:tasks.ceph.osd.3:Stopped 2026-03-31T21:27:56.394 INFO:teuthology.misc:Shutting down mgr daemons... 2026-03-31T21:27:56.394 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-03-31T21:27:56.394 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.431 INFO:tasks.ceph.mgr.x:Stopped 2026-03-31T21:27:56.431 INFO:teuthology.misc:Shutting down mon daemons... 2026-03-31T21:27:56.431 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-03-31T21:27:56.431 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.484 INFO:tasks.ceph.mon.a:Stopped 2026-03-31T21:27:56.484 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-03-31T21:27:56.484 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.535 INFO:tasks.ceph.mon.b:Stopped 2026-03-31T21:27:56.535 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-03-31T21:27:56.535 INFO:teuthology.orchestra.run:waiting for 300 2026-03-31T21:27:56.586 INFO:tasks.ceph.mon.c:Stopped 2026-03-31T21:27:56.586 INFO:tasks.ceph:Checking cluster log for badness... 2026-03-31T21:27:56.586 DEBUG:teuthology.orchestra.run.vm00:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(OBJECT_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SLOW_OPS\)' | egrep -v deep-scrub | egrep -v missing | egrep -v 'overall HEALTH_' | egrep -v repair | egrep -v 'slow request' | egrep -v unfound | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'enough copies available' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2026-03-31T21:27:56.656 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@vm00.local 2026-03-31T21:27:56.656 DEBUG:teuthology.orchestra.run.vm00:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-03-31T21:27:56.736 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@vm00.local 2026-03-31T21:27:56.736 DEBUG:teuthology.orchestra.run.vm00:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-03-31T21:27:56.810 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@vm00.local 2026-03-31T21:27:56.810 DEBUG:teuthology.orchestra.run.vm00:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-03-31T21:27:56.880 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@vm00.local 2026-03-31T21:27:56.880 DEBUG:teuthology.orchestra.run.vm00:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-03-31T21:27:56.962 INFO:tasks.ceph:Archiving mon data... 2026-03-31T21:27:56.963 DEBUG:teuthology.misc:Transferring archived files from vm00:/var/lib/ceph/mon/ceph-a to /archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348/data/mon.a.tgz 2026-03-31T21:27:56.963 DEBUG:teuthology.orchestra.run.vm00:> mktemp 2026-03-31T21:27:57.019 INFO:teuthology.orchestra.run.vm00.stdout:/tmp/tmp.bFB9dIwn4u 2026-03-31T21:27:57.019 DEBUG:teuthology.orchestra.run.vm00:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bFB9dIwn4u 2026-03-31T21:27:57.120 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod 0666 /tmp/tmp.bFB9dIwn4u 2026-03-31T21:27:57.201 DEBUG:teuthology.orchestra.remote:vm00:/tmp/tmp.bFB9dIwn4u is 314KB 2026-03-31T21:27:57.256 DEBUG:teuthology.orchestra.run.vm00:> rm -fr /tmp/tmp.bFB9dIwn4u 2026-03-31T21:27:57.271 DEBUG:teuthology.misc:Transferring archived files from vm00:/var/lib/ceph/mon/ceph-b to /archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348/data/mon.b.tgz 2026-03-31T21:27:57.271 DEBUG:teuthology.orchestra.run.vm00:> mktemp 2026-03-31T21:27:57.324 INFO:teuthology.orchestra.run.vm00.stdout:/tmp/tmp.kEdqhr6RDh 2026-03-31T21:27:57.324 DEBUG:teuthology.orchestra.run.vm00:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.kEdqhr6RDh 2026-03-31T21:27:57.427 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod 0666 /tmp/tmp.kEdqhr6RDh 2026-03-31T21:27:57.509 DEBUG:teuthology.orchestra.remote:vm00:/tmp/tmp.kEdqhr6RDh is 314KB 2026-03-31T21:27:57.567 DEBUG:teuthology.orchestra.run.vm00:> rm -fr /tmp/tmp.kEdqhr6RDh 2026-03-31T21:27:57.581 DEBUG:teuthology.misc:Transferring archived files from vm00:/var/lib/ceph/mon/ceph-c to /archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348/data/mon.c.tgz 2026-03-31T21:27:57.581 DEBUG:teuthology.orchestra.run.vm00:> mktemp 2026-03-31T21:27:57.637 INFO:teuthology.orchestra.run.vm00.stdout:/tmp/tmp.iuxbj9QFqk 2026-03-31T21:27:57.637 DEBUG:teuthology.orchestra.run.vm00:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.iuxbj9QFqk 2026-03-31T21:27:57.738 DEBUG:teuthology.orchestra.run.vm00:> sudo chmod 0666 /tmp/tmp.iuxbj9QFqk 2026-03-31T21:27:57.813 DEBUG:teuthology.orchestra.remote:vm00:/tmp/tmp.iuxbj9QFqk is 314KB 2026-03-31T21:27:57.868 DEBUG:teuthology.orchestra.run.vm00:> rm -fr /tmp/tmp.iuxbj9QFqk 2026-03-31T21:27:57.880 INFO:tasks.ceph:Cleaning ceph cluster... 2026-03-31T21:27:57.880 DEBUG:teuthology.orchestra.run.vm00:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-03-31T21:27:57.997 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-03-31T21:27:57.997 INFO:tasks.ceph:Archiving crash dumps... 2026-03-31T21:27:57.997 DEBUG:teuthology.misc:Transferring archived files from vm00:/var/lib/ceph/crash to /archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348/remote/vm00/crash 2026-03-31T21:27:57.997 DEBUG:teuthology.orchestra.run.vm00:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-03-31T21:27:58.059 INFO:tasks.ceph:Compressing logs... 2026-03-31T21:27:58.059 DEBUG:teuthology.orchestra.run.vm00:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-03-31T21:27:58.079 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.53051.log 2026-03-31T21:27:58.079 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-03-31T21:27:58.079 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph.tmp-client.admin.53051.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.53051.log.gz 2026-03-31T21:27:58.079 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-03-31T21:27:58.080 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-03-31T21:27:58.080 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-03-31T21:27:58.080 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-03-31T21:27:58.095 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-03-31T21:27:58.100 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-03-31T21:27:58.114 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59226.log 2026-03-31T21:27:58.119 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-03-31T21:27:58.120 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59226.log.gz 2026-03-31T21:27:58.135 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-03-31T21:27:58.140 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph.log: 95.8% -- replaced with /var/log/ceph/ceph.log.gz 2026-03-31T21:27:58.154 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59416.log 2026-03-31T21:27:58.160 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59486.log 2026-03-31T21:27:58.161 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59416.log.gz 2026-03-31T21:27:58.172 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59644.log 2026-03-31T21:27:58.173 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59486.log.gz 2026-03-31T21:27:58.180 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59691.log 2026-03-31T21:27:58.181 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59644.log.gz 2026-03-31T21:27:58.190 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-03-31T21:27:58.191 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59691.log.gz 2026-03-31T21:27:58.199 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59738.log 2026-03-31T21:27:58.205 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph.audit.log: 96.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-03-31T21:27:58.210 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59785.log 2026-03-31T21:27:58.211 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59738.log.gz 2026-03-31T21:27:58.219 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59940.log 2026-03-31T21:27:58.219 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59785.log.gz 2026-03-31T21:27:58.230 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62415.log 2026-03-31T21:27:58.231 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.59940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59940.log.gz 2026-03-31T21:27:58.238 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63380.log 2026-03-31T21:27:58.238 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.62415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62415.log.gz 2026-03-31T21:27:58.249 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63452.log 2026-03-31T21:27:58.249 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63380.log.gz 2026-03-31T21:27:58.257 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63501.log 2026-03-31T21:27:58.257 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63452.log.gz 2026-03-31T21:27:58.268 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63550.log 2026-03-31T21:27:58.268 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63501.log.gz 2026-03-31T21:27:58.278 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63599.log 2026-03-31T21:27:58.278 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63550.log: 29.6% -- replaced with /var/log/ceph/ceph-client.admin.63550.log.gz 2026-03-31T21:27:58.287 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63724.log 2026-03-31T21:27:58.287 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63599.log.gz 2026-03-31T21:27:58.296 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63729.log 2026-03-31T21:27:58.297 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63724.log.gz 2026-03-31T21:27:58.305 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63727.log 2026-03-31T21:27:58.306 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63729.log.gz 2026-03-31T21:27:58.322 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63728.log 2026-03-31T21:27:58.323 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63727.log.gz 2026-03-31T21:27:58.338 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63870.log 2026-03-31T21:27:58.339 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63728.log.gz 2026-03-31T21:27:58.344 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63871.log 2026-03-31T21:27:58.344 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63870.log.gz 2026-03-31T21:27:58.355 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63963.log 2026-03-31T21:27:58.356 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63871.log.gz 2026-03-31T21:27:58.363 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63961.log 2026-03-31T21:27:58.363 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63963.log.gz 2026-03-31T21:27:58.375 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64040.log 2026-03-31T21:27:58.376 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.63961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63961.log.gz 2026-03-31T21:27:58.383 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64067.log 2026-03-31T21:27:58.383 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64040.log.gz 2026-03-31T21:27:58.392 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64114.log 2026-03-31T21:27:58.393 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64067.log.gz 2026-03-31T21:27:58.393 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64163.log 2026-03-31T21:27:58.393 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64114.log.gz 2026-03-31T21:27:58.408 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64237.log 2026-03-31T21:27:58.408 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64163.log.gz 2026-03-31T21:27:58.424 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64286.log 2026-03-31T21:27:58.424 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64237.log.gz 2026-03-31T21:27:58.439 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64335.log 2026-03-31T21:27:58.439 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64286.log.gz 2026-03-31T21:27:58.448 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64384.log 2026-03-31T21:27:58.448 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64335.log.gz 2026-03-31T21:27:58.454 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64433.log 2026-03-31T21:27:58.455 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64384.log.gz 2026-03-31T21:27:58.469 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64482.log 2026-03-31T21:27:58.469 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64433.log.gz 2026-03-31T21:27:58.476 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64531.log 2026-03-31T21:27:58.476 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64482.log.gz 2026-03-31T21:27:58.494 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64582.log 2026-03-31T21:27:58.494 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64531.log.gz 2026-03-31T21:27:58.511 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64632.log 2026-03-31T21:27:58.511 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64582.log.gz 2026-03-31T21:27:58.513 INFO:teuthology.orchestra.run.vm00.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-03-31T21:27:58.524 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64681.log 2026-03-31T21:27:58.525 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64632.log.gz 2026-03-31T21:27:58.536 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64809.log 2026-03-31T21:27:58.537 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64681.log.gz 2026-03-31T21:27:58.550 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64810.log 2026-03-31T21:27:58.551 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64809.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64809.log.gz 2026-03-31T21:27:58.568 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64806.log 2026-03-31T21:27:58.569 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64810.log.gz 2026-03-31T21:27:58.580 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64811.log 2026-03-31T21:27:58.581 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64806.log.gz 2026-03-31T21:27:58.592 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64959.log 2026-03-31T21:27:58.593 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64811.log.gz 2026-03-31T21:27:58.606 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64976.log 2026-03-31T21:27:58.607 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64959.log.gz 2026-03-31T21:27:58.620 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64980.log 2026-03-31T21:27:58.621 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64976.log.gz 2026-03-31T21:27:58.634 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65034.log 2026-03-31T21:27:58.635 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.64980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64980.log.gz 2026-03-31T21:27:58.648 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65122.log 2026-03-31T21:27:58.649 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65034.log.gz 2026-03-31T21:27:58.662 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65198.log 2026-03-31T21:27:58.663 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65122.log.gz 2026-03-31T21:27:58.676 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65204.log 2026-03-31T21:27:58.677 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65198.log.gz 2026-03-31T21:27:58.691 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65203.log 2026-03-31T21:27:58.692 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65204.log.gz 2026-03-31T21:27:58.712 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65318.log 2026-03-31T21:27:58.713 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65203.log: 28.3% -- replaced with /var/log/ceph/ceph-client.admin.65203.log.gz 2026-03-31T21:27:58.726 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65370.log 2026-03-31T21:27:58.727 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65318.log.gz 2026-03-31T21:27:58.740 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65373.log 2026-03-31T21:27:58.741 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65370.log.gz 2026-03-31T21:27:58.754 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65465.log 2026-03-31T21:27:58.755 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65373.log.gz 2026-03-31T21:27:58.774 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65514.log 2026-03-31T21:27:58.775 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65465.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65465.log.gz 2026-03-31T21:27:58.786 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65563.log 2026-03-31T21:27:58.787 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65514.log.gz 2026-03-31T21:27:58.799 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65612.log 2026-03-31T21:27:58.800 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65563.log.gz 2026-03-31T21:27:58.813 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65661.log 2026-03-31T21:27:58.814 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65612.log.gz 2026-03-31T21:27:58.818 INFO:teuthology.orchestra.run.vm00.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-03-31T21:27:58.832 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65684.log 2026-03-31T21:27:58.833 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65661.log.gz 2026-03-31T21:27:58.841 INFO:teuthology.orchestra.run.vm00.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-03-31T21:27:58.847 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65780.log 2026-03-31T21:27:58.847 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65684.log.gz 2026-03-31T21:27:58.854 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65781.log 2026-03-31T21:27:58.855 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65780.log.gz 2026-03-31T21:27:58.863 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65878.log 2026-03-31T21:27:58.863 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65781.log.gz 2026-03-31T21:27:58.873 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65879.log 2026-03-31T21:27:58.874 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65878.log.gz 2026-03-31T21:27:58.880 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65950.log 2026-03-31T21:27:58.880 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65879.log.gz 2026-03-31T21:27:58.888 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65977.log 2026-03-31T21:27:58.889 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65950.log.gz 2026-03-31T21:27:58.898 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66048.log 2026-03-31T21:27:58.899 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.65977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65977.log.gz 2026-03-31T21:27:58.907 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66075.log 2026-03-31T21:27:58.907 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.66048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66048.log.gz 2026-03-31T21:27:58.915 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66146.log 2026-03-31T21:27:58.915 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.66075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66075.log.gz 2026-03-31T21:27:58.922 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66195.log 2026-03-31T21:27:58.922 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.66146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66146.log.gz 2026-03-31T21:27:58.930 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66218.log 2026-03-31T21:27:58.931 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.66195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66195.log.gz 2026-03-31T21:27:58.939 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66288.log 2026-03-31T21:27:58.939 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.66218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66218.log.gz 2026-03-31T21:27:58.946 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66337.log 2026-03-31T21:27:58.946 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.66288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66288.log.gz 2026-03-31T21:27:58.961 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68289.log 2026-03-31T21:27:58.961 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.66337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66337.log.gz 2026-03-31T21:27:58.976 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68338.log 2026-03-31T21:27:58.976 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68289.log.gz 2026-03-31T21:27:58.991 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68466.log 2026-03-31T21:27:58.991 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68338.log.gz 2026-03-31T21:27:59.005 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68467.log 2026-03-31T21:27:59.005 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68466.log.gz 2026-03-31T21:27:59.020 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68468.log 2026-03-31T21:27:59.020 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68467.log.gz 2026-03-31T21:27:59.034 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68464.log 2026-03-31T21:27:59.034 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68468.log.gz 2026-03-31T21:27:59.051 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68581.log 2026-03-31T21:27:59.051 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68464.log.gz 2026-03-31T21:27:59.058 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68636.log 2026-03-31T21:27:59.058 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68581.log.gz 2026-03-31T21:27:59.067 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68645.log 2026-03-31T21:27:59.068 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68636.log.gz 2026-03-31T21:27:59.076 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68726.log 2026-03-31T21:27:59.077 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68645.log.gz 2026-03-31T21:27:59.083 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68779.log 2026-03-31T21:27:59.084 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68726.log.gz 2026-03-31T21:27:59.090 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68828.log 2026-03-31T21:27:59.091 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68779.log.gz 2026-03-31T21:27:59.106 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68877.log 2026-03-31T21:27:59.107 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68828.log.gz 2026-03-31T21:27:59.114 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68926.log 2026-03-31T21:27:59.115 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68877.log.gz 2026-03-31T21:27:59.121 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69056.log 2026-03-31T21:27:59.122 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.68926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68926.log.gz 2026-03-31T21:27:59.129 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69054.log 2026-03-31T21:27:59.130 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69056.log.gz 2026-03-31T21:27:59.136 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69053.log 2026-03-31T21:27:59.137 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69054.log.gz 2026-03-31T21:27:59.143 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69055.log 2026-03-31T21:27:59.144 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69053.log.gz 2026-03-31T21:27:59.151 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69240.log 2026-03-31T21:27:59.151 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69055.log.gz 2026-03-31T21:27:59.159 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69250.log 2026-03-31T21:27:59.160 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69240.log.gz 2026-03-31T21:27:59.168 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69252.log 2026-03-31T21:27:59.168 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69250.log.gz 2026-03-31T21:27:59.175 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69251.log 2026-03-31T21:27:59.175 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69252.log.gz 2026-03-31T21:27:59.181 INFO:teuthology.orchestra.run.vm00.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-03-31T21:27:59.181 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69367.log 2026-03-31T21:27:59.181 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69251.log.gz 2026-03-31T21:27:59.182 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69443.log 2026-03-31T21:27:59.182 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69367.log.gz 2026-03-31T21:27:59.182 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69448.log 2026-03-31T21:27:59.182 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69443.log: 29.4% -- replaced with /var/log/ceph/ceph-client.admin.69443.log.gz 2026-03-31T21:27:59.183 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69458.log 2026-03-31T21:27:59.183 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69448.log.gz 2026-03-31T21:27:59.183 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69563.log 2026-03-31T21:27:59.183 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69458.log.gz 2026-03-31T21:27:59.184 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69617.log 2026-03-31T21:27:59.184 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69563.log.gz 2026-03-31T21:27:59.184 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69616.log 2026-03-31T21:27:59.184 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69617.log.gz 2026-03-31T21:27:59.184 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69706.log 2026-03-31T21:27:59.185 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69616.log.gz 2026-03-31T21:27:59.185 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69759.log 2026-03-31T21:27:59.185 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69706.log.gz 2026-03-31T21:27:59.185 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69782.log 2026-03-31T21:27:59.186 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69759.log.gz 2026-03-31T21:27:59.186 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69853.log 2026-03-31T21:27:59.186 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69782.log: 95.1% -- replaced with /var/log/ceph/ceph-client.admin.69782.log.gz 2026-03-31T21:27:59.186 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69902.log 2026-03-31T21:27:59.186 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69853.log.gz 2026-03-31T21:27:59.187 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71889.log 2026-03-31T21:27:59.187 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.69902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69902.log.gz 2026-03-31T21:27:59.187 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71938.log 2026-03-31T21:27:59.187 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.71889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71889.log.gz 2026-03-31T21:27:59.188 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72067.log 2026-03-31T21:27:59.188 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.71938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71938.log.gz 2026-03-31T21:27:59.188 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72066.log 2026-03-31T21:27:59.188 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72067.log.gz 2026-03-31T21:27:59.189 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72064.log 2026-03-31T21:27:59.189 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72066.log.gz 2026-03-31T21:27:59.189 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72068.log 2026-03-31T21:27:59.189 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72064.log.gz 2026-03-31T21:27:59.190 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72217.log 2026-03-31T21:27:59.190 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72068.log.gz 2026-03-31T21:27:59.190 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72236.log 2026-03-31T21:27:59.190 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72217.log.gz 2026-03-31T21:27:59.190 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72237.log 2026-03-31T21:27:59.191 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72236.log.gz 2026-03-31T21:27:59.191 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72287.log 2026-03-31T21:27:59.191 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72237.log.gz 2026-03-31T21:27:59.191 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72406.log 2026-03-31T21:27:59.191 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72287.log.gz 2026-03-31T21:27:59.192 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72404.log 2026-03-31T21:27:59.192 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72406.log.gz 2026-03-31T21:27:59.192 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72434.log 2026-03-31T21:27:59.192 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72404.log.gz 2026-03-31T21:27:59.193 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72500.log 2026-03-31T21:27:59.193 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72434.log.gz 2026-03-31T21:27:59.193 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72575.log 2026-03-31T21:27:59.193 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72500.log.gz 2026-03-31T21:27:59.194 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72704.log 2026-03-31T21:27:59.194 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72575.log.gz 2026-03-31T21:27:59.194 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72698.log 2026-03-31T21:27:59.194 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72704.log.gz 2026-03-31T21:27:59.194 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72705.log 2026-03-31T21:27:59.195 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72698.log.gz 2026-03-31T21:27:59.195 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72699.log 2026-03-31T21:27:59.195 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72705.log.gz 2026-03-31T21:27:59.195 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72899.log 2026-03-31T21:27:59.195 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72699.log.gz 2026-03-31T21:27:59.196 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72885.log 2026-03-31T21:27:59.196 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72899.log.gz 2026-03-31T21:27:59.196 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72901.log 2026-03-31T21:27:59.197 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72885.log.gz 2026-03-31T21:27:59.197 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72900.log 2026-03-31T21:27:59.197 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72901.log.gz 2026-03-31T21:27:59.197 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73016.log 2026-03-31T21:27:59.197 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.72900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72900.log.gz 2026-03-31T21:27:59.198 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73094.log 2026-03-31T21:27:59.198 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73016.log.gz 2026-03-31T21:27:59.198 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73097.log 2026-03-31T21:27:59.198 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73094.log.gz 2026-03-31T21:27:59.199 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73096.log 2026-03-31T21:27:59.199 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73097.log.gz 2026-03-31T21:27:59.199 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73212.log 2026-03-31T21:27:59.199 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73096.log.gz 2026-03-31T21:27:59.200 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73366.log 2026-03-31T21:27:59.200 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73212.log.gz 2026-03-31T21:27:59.200 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73374.log 2026-03-31T21:27:59.200 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73366.log.gz 2026-03-31T21:27:59.200 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73370.log 2026-03-31T21:27:59.201 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73374.log.gz 2026-03-31T21:27:59.201 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73371.log 2026-03-31T21:27:59.201 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73370.log.gz 2026-03-31T21:27:59.201 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73487.log 2026-03-31T21:27:59.201 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73371.log.gz 2026-03-31T21:27:59.202 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73543.log 2026-03-31T21:27:59.202 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73487.log.gz 2026-03-31T21:27:59.202 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73540.log 2026-03-31T21:27:59.202 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73543.log.gz 2026-03-31T21:27:59.203 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73594.log 2026-03-31T21:27:59.203 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73540.log.gz 2026-03-31T21:27:59.203 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73685.log 2026-03-31T21:27:59.203 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73594.log.gz 2026-03-31T21:27:59.203 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73712.log 2026-03-31T21:27:59.204 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73685.log.gz 2026-03-31T21:27:59.204 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73759.log 2026-03-31T21:27:59.204 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73712.log.gz 2026-03-31T21:27:59.204 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73806.log 2026-03-31T21:27:59.204 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73759.log.gz 2026-03-31T21:27:59.205 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73886.log 2026-03-31T21:27:59.205 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73806.log.gz 2026-03-31T21:27:59.205 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74013.log 2026-03-31T21:27:59.205 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.73886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73886.log.gz 2026-03-31T21:27:59.206 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74016.log 2026-03-31T21:27:59.206 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74013.log.gz 2026-03-31T21:27:59.206 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74012.log 2026-03-31T21:27:59.206 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74016.log.gz 2026-03-31T21:27:59.207 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74015.log 2026-03-31T21:27:59.207 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74012.log.gz 2026-03-31T21:27:59.207 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74131.log 2026-03-31T21:27:59.207 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74015.log.gz 2026-03-31T21:27:59.207 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74204.log 2026-03-31T21:27:59.207 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74131.log.gz 2026-03-31T21:27:59.208 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74205.log 2026-03-31T21:27:59.208 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74204.log.gz 2026-03-31T21:27:59.208 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74232.log 2026-03-31T21:27:59.208 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74205.log.gz 2026-03-31T21:27:59.209 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74327.log 2026-03-31T21:27:59.209 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74232.log.gz 2026-03-31T21:27:59.209 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74374.log 2026-03-31T21:27:59.209 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74327.log.gz 2026-03-31T21:27:59.210 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74429.log 2026-03-31T21:27:59.210 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74374.log.gz 2026-03-31T21:27:59.210 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74428.log 2026-03-31T21:27:59.210 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74429.log.gz 2026-03-31T21:27:59.210 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74523.log 2026-03-31T21:27:59.211 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74428.log.gz 2026-03-31T21:27:59.211 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74657.log 2026-03-31T21:27:59.211 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74523.log.gz 2026-03-31T21:27:59.211 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74656.log 2026-03-31T21:27:59.212 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74657.log.gz 2026-03-31T21:27:59.212 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74655.log 2026-03-31T21:27:59.212 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74656.log.gz 2026-03-31T21:27:59.212 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74658.log 2026-03-31T21:27:59.213 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74655.log.gz 2026-03-31T21:27:59.213 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74797.log 2026-03-31T21:27:59.213 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74658.log.gz 2026-03-31T21:27:59.213 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74800.log 2026-03-31T21:27:59.213 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74797.log.gz 2026-03-31T21:27:59.214 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74854.log 2026-03-31T21:27:59.214 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74800.log: 28.3% -- replaced with /var/log/ceph/ceph-client.admin.74800.log.gz 2026-03-31T21:27:59.214 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74853.log 2026-03-31T21:27:59.214 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74854.log.gz 2026-03-31T21:27:59.215 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74969.log 2026-03-31T21:27:59.215 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74853.log.gz 2026-03-31T21:27:59.215 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75098.log 2026-03-31T21:27:59.215 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.74969.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.74969.log.gz 2026-03-31T21:27:59.215 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75097.log 2026-03-31T21:27:59.216 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75098.log.gz 2026-03-31T21:27:59.216 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75095.log 2026-03-31T21:27:59.216 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75097.log.gz 2026-03-31T21:27:59.216 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75099.log 2026-03-31T21:27:59.216 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75095.log.gz 2026-03-31T21:27:59.217 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75214.log 2026-03-31T21:27:59.217 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75099.log.gz 2026-03-31T21:27:59.217 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75315.log 2026-03-31T21:27:59.217 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75214.log.gz 2026-03-31T21:27:59.218 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75310.log 2026-03-31T21:27:59.218 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75315.log.gz 2026-03-31T21:27:59.218 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75313.log 2026-03-31T21:27:59.218 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75310.log.gz 2026-03-31T21:27:59.218 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75410.log 2026-03-31T21:27:59.219 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75313.log.gz 2026-03-31T21:27:59.219 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75538.log 2026-03-31T21:27:59.219 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75410.log.gz 2026-03-31T21:27:59.219 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75536.log 2026-03-31T21:27:59.220 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75538.log.gz 2026-03-31T21:27:59.220 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75542.log 2026-03-31T21:27:59.220 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75536.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75536.log.gz 2026-03-31T21:27:59.220 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75539.log 2026-03-31T21:27:59.220 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75542.log.gz 2026-03-31T21:27:59.221 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75657.log 2026-03-31T21:27:59.221 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75539.log.gz 2026-03-31T21:27:59.221 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75710.log 2026-03-31T21:27:59.221 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75657.log.gz 2026-03-31T21:27:59.222 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75711.log 2026-03-31T21:27:59.222 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75710.log.gz 2026-03-31T21:27:59.222 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75772.log 2026-03-31T21:27:59.222 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75711.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75711.log.gz 2026-03-31T21:27:59.222 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75853.log 2026-03-31T21:27:59.223 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75772.log.gz 2026-03-31T21:27:59.223 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75902.log 2026-03-31T21:27:59.223 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75853.log.gz 2026-03-31T21:27:59.223 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76030.log 2026-03-31T21:27:59.223 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.75902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.75902.log.gz 2026-03-31T21:27:59.224 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76029.log 2026-03-31T21:27:59.224 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76030.log.gz 2026-03-31T21:27:59.224 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76027.log 2026-03-31T21:27:59.224 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76029.log.gz 2026-03-31T21:27:59.225 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76032.log 2026-03-31T21:27:59.225 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76027.log.gz 2026-03-31T21:27:59.225 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76148.log 2026-03-31T21:27:59.225 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76032.log.gz 2026-03-31T21:27:59.226 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76200.log 2026-03-31T21:27:59.226 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76148.log.gz 2026-03-31T21:27:59.226 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76201.log 2026-03-31T21:27:59.226 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76200.log.gz 2026-03-31T21:27:59.227 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76260.log 2026-03-31T21:27:59.227 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76201.log.gz 2026-03-31T21:27:59.227 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76344.log 2026-03-31T21:27:59.227 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76260.log.gz 2026-03-31T21:27:59.227 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76398.log 2026-03-31T21:27:59.228 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76344.log.gz 2026-03-31T21:27:59.228 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76397.log 2026-03-31T21:27:59.228 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76398.log.gz 2026-03-31T21:27:59.228 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76491.log 2026-03-31T21:27:59.228 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76397.log.gz 2026-03-31T21:27:59.229 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76540.log 2026-03-31T21:27:59.229 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76491.log.gz 2026-03-31T21:27:59.229 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76574.log 2026-03-31T21:27:59.229 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76540.log.gz 2026-03-31T21:27:59.230 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76614.log 2026-03-31T21:27:59.230 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76574.log.gz 2026-03-31T21:27:59.230 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76687.log 2026-03-31T21:27:59.230 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76614.log.gz 2026-03-31T21:27:59.230 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76736.log 2026-03-31T21:27:59.231 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76687.log.gz 2026-03-31T21:27:59.231 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76785.log 2026-03-31T21:27:59.231 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76736.log.gz 2026-03-31T21:27:59.231 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76914.log 2026-03-31T21:27:59.232 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76785.log.gz 2026-03-31T21:27:59.232 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76912.log 2026-03-31T21:27:59.232 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76914.log.gz 2026-03-31T21:27:59.232 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76915.log 2026-03-31T21:27:59.232 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76912.log.gz 2026-03-31T21:27:59.233 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76913.log 2026-03-31T21:27:59.233 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76915.log.gz 2026-03-31T21:27:59.233 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77091.log 2026-03-31T21:27:59.233 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.76913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.76913.log.gz 2026-03-31T21:27:59.234 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77110.log 2026-03-31T21:27:59.234 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77091.log.gz 2026-03-31T21:27:59.234 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77111.log 2026-03-31T21:27:59.234 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77110.log.gz 2026-03-31T21:27:59.234 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77109.log 2026-03-31T21:27:59.235 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77111.log.gz 2026-03-31T21:27:59.235 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77226.log 2026-03-31T21:27:59.235 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77109.log.gz 2026-03-31T21:27:59.235 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77308.log 2026-03-31T21:27:59.235 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77226.log.gz 2026-03-31T21:27:59.236 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77306.log 2026-03-31T21:27:59.236 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77308.log.gz 2026-03-31T21:27:59.236 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77307.log 2026-03-31T21:27:59.236 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77306.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77306.log.gz 2026-03-31T21:27:59.237 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77422.log 2026-03-31T21:27:59.237 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77307.log.gz 2026-03-31T21:27:59.237 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77509.log 2026-03-31T21:27:59.237 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77422.log.gz 2026-03-31T21:27:59.237 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77521.log 2026-03-31T21:27:59.238 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77509.log.gz 2026-03-31T21:27:59.238 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77511.log 2026-03-31T21:27:59.238 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77521.log.gz 2026-03-31T21:27:59.238 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77618.log 2026-03-31T21:27:59.238 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77511.log.gz 2026-03-31T21:27:59.239 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77667.log 2026-03-31T21:27:59.239 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77618.log.gz 2026-03-31T21:27:59.239 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77716.log 2026-03-31T21:27:59.239 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77667.log.gz 2026-03-31T21:27:59.240 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77765.log 2026-03-31T21:27:59.240 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77716.log.gz 2026-03-31T21:27:59.240 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79718.log 2026-03-31T21:27:59.240 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.77765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.77765.log.gz 2026-03-31T21:27:59.240 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79767.log 2026-03-31T21:27:59.241 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.79718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79718.log.gz 2026-03-31T21:27:59.241 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79895.log 2026-03-31T21:27:59.241 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.79767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79767.log.gz 2026-03-31T21:27:59.241 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79893.log 2026-03-31T21:27:59.242 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.79895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79895.log.gz 2026-03-31T21:27:59.242 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79892.log 2026-03-31T21:27:59.242 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.79893.log: 30.9% -- replaced with /var/log/ceph/ceph-client.admin.79893.log.gz 2026-03-31T21:27:59.242 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79897.log 2026-03-31T21:27:59.242 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.79892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79892.log.gz 2026-03-31T21:27:59.243 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80008.log 2026-03-31T21:27:59.243 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.79897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.79897.log.gz 2026-03-31T21:27:59.243 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80068.log 2026-03-31T21:27:59.243 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80008.log.gz 2026-03-31T21:27:59.244 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80069.log 2026-03-31T21:27:59.244 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80068.log.gz 2026-03-31T21:27:59.244 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80154.log 2026-03-31T21:27:59.244 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80069.log.gz 2026-03-31T21:27:59.245 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80208.log 2026-03-31T21:27:59.245 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80154.log.gz 2026-03-31T21:27:59.245 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80260.log 2026-03-31T21:27:59.245 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80208.log.gz 2026-03-31T21:27:59.245 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80263.log 2026-03-31T21:27:59.246 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80260.log.gz 2026-03-31T21:27:59.246 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80349.log 2026-03-31T21:27:59.246 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80263.log.gz 2026-03-31T21:27:59.246 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80404.log 2026-03-31T21:27:59.246 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80349.log: 29.6% -- replaced with /var/log/ceph/ceph-client.admin.80349.log.gz 2026-03-31T21:27:59.247 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80458.log 2026-03-31T21:27:59.247 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80404.log.gz 2026-03-31T21:27:59.247 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80459.log 2026-03-31T21:27:59.247 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80458.log.gz 2026-03-31T21:27:59.248 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80551.log 2026-03-31T21:27:59.248 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80459.log.gz 2026-03-31T21:27:59.248 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80600.log 2026-03-31T21:27:59.248 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80551.log.gz 2026-03-31T21:27:59.249 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80726.log 2026-03-31T21:27:59.249 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80600.log.gz 2026-03-31T21:27:59.249 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80724.log 2026-03-31T21:27:59.249 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80726.log.gz 2026-03-31T21:27:59.249 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80728.log 2026-03-31T21:27:59.250 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80724.log.gz 2026-03-31T21:27:59.250 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80730.log 2026-03-31T21:27:59.250 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80728.log.gz 2026-03-31T21:27:59.250 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80845.log 2026-03-31T21:27:59.250 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80730.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80730.log.gz 2026-03-31T21:27:59.251 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80919.log 2026-03-31T21:27:59.251 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80845.log.gz 2026-03-31T21:27:59.251 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80918.log 2026-03-31T21:27:59.251 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80919.log.gz 2026-03-31T21:27:59.252 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80950.log 2026-03-31T21:27:59.252 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80918.log.gz 2026-03-31T21:27:59.252 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81041.log 2026-03-31T21:27:59.252 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.80950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.80950.log.gz 2026-03-31T21:27:59.253 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81115.log 2026-03-31T21:27:59.253 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81041.log.gz 2026-03-31T21:27:59.253 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81112.log 2026-03-31T21:27:59.253 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81115.log.gz 2026-03-31T21:27:59.253 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81150.log 2026-03-31T21:27:59.254 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81112.log.gz 2026-03-31T21:27:59.254 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81237.log 2026-03-31T21:27:59.254 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81150.log.gz 2026-03-31T21:27:59.254 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81284.log 2026-03-31T21:27:59.254 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81237.log.gz 2026-03-31T21:27:59.255 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81331.log 2026-03-31T21:27:59.255 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81284.log.gz 2026-03-31T21:27:59.255 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81384.log 2026-03-31T21:27:59.255 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81331.log.gz 2026-03-31T21:27:59.256 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81433.log 2026-03-31T21:27:59.256 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81384.log.gz 2026-03-31T21:27:59.256 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81482.log 2026-03-31T21:27:59.256 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81433.log.gz 2026-03-31T21:27:59.257 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81531.log 2026-03-31T21:27:59.257 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81482.log.gz 2026-03-31T21:27:59.257 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81654.log 2026-03-31T21:27:59.257 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81531.log.gz 2026-03-31T21:27:59.258 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81657.log 2026-03-31T21:27:59.258 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81654.log.gz 2026-03-31T21:27:59.258 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81660.log 2026-03-31T21:27:59.258 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81657.log.gz 2026-03-31T21:27:59.258 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81661.log 2026-03-31T21:27:59.259 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81660.log.gz 2026-03-31T21:27:59.259 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81845.log 2026-03-31T21:27:59.259 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81661.log.gz 2026-03-31T21:27:59.259 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81855.log 2026-03-31T21:27:59.259 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81845.log.gz 2026-03-31T21:27:59.260 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81857.log 2026-03-31T21:27:59.260 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81855.log.gz 2026-03-31T21:27:59.260 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81856.log 2026-03-31T21:27:59.260 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81857.log.gz 2026-03-31T21:27:59.261 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81972.log 2026-03-31T21:27:59.261 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81856.log.gz 2026-03-31T21:27:59.261 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82053.log 2026-03-31T21:27:59.261 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.81972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.81972.log.gz 2026-03-31T21:27:59.261 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82048.log 2026-03-31T21:27:59.262 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82053.log.gz 2026-03-31T21:27:59.262 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82054.log 2026-03-31T21:27:59.262 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82048.log.gz 2026-03-31T21:27:59.262 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82168.log 2026-03-31T21:27:59.263 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82054.log.gz 2026-03-31T21:27:59.263 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82196.log 2026-03-31T21:27:59.263 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82168.log.gz 2026-03-31T21:27:59.263 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82265.log 2026-03-31T21:27:59.263 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82196.log.gz 2026-03-31T21:27:59.264 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82270.log 2026-03-31T21:27:59.264 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82265.log.gz 2026-03-31T21:27:59.264 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82364.log 2026-03-31T21:27:59.264 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82270.log.gz 2026-03-31T21:27:59.265 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82413.log 2026-03-31T21:27:59.265 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82364.log.gz 2026-03-31T21:27:59.265 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82462.log 2026-03-31T21:27:59.265 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82413.log.gz 2026-03-31T21:27:59.265 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82511.log 2026-03-31T21:27:59.266 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82462.log.gz 2026-03-31T21:27:59.266 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84464.log 2026-03-31T21:27:59.266 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.82511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.82511.log.gz 2026-03-31T21:27:59.266 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84513.log 2026-03-31T21:27:59.266 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.84464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84464.log.gz 2026-03-31T21:27:59.267 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84562.log 2026-03-31T21:27:59.267 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.84513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84513.log.gz 2026-03-31T21:27:59.267 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86515.log 2026-03-31T21:27:59.267 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.84562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.84562.log.gz 2026-03-31T21:27:59.268 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86564.log 2026-03-31T21:27:59.268 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86515.log.gz 2026-03-31T21:27:59.268 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86689.log 2026-03-31T21:27:59.268 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86564.log.gz 2026-03-31T21:27:59.269 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86688.log 2026-03-31T21:27:59.269 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86689.log.gz 2026-03-31T21:27:59.269 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86694.log 2026-03-31T21:27:59.269 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86688.log.gz 2026-03-31T21:27:59.269 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86693.log 2026-03-31T21:27:59.270 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86694.log.gz 2026-03-31T21:27:59.270 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86890.log 2026-03-31T21:27:59.270 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86693.log.gz 2026-03-31T21:27:59.270 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86887.log 2026-03-31T21:27:59.270 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86890.log.gz 2026-03-31T21:27:59.271 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86889.log 2026-03-31T21:27:59.271 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86887.log.gz 2026-03-31T21:27:59.271 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86888.log 2026-03-31T21:27:59.271 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86889.log.gz 2026-03-31T21:27:59.272 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87031.log 2026-03-31T21:27:59.272 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.86888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.86888.log.gz 2026-03-31T21:27:59.272 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87032.log 2026-03-31T21:27:59.272 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87031.log.gz 2026-03-31T21:27:59.273 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87087.log 2026-03-31T21:27:59.273 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87032.log.gz 2026-03-31T21:27:59.273 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87116.log 2026-03-31T21:27:59.273 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87087.log.gz 2026-03-31T21:27:59.273 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87201.log 2026-03-31T21:27:59.274 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87116.log.gz 2026-03-31T21:27:59.274 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87330.log 2026-03-31T21:27:59.274 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87201.log.gz 2026-03-31T21:27:59.274 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87331.log 2026-03-31T21:27:59.275 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87330.log.gz 2026-03-31T21:27:59.275 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87329.log 2026-03-31T21:27:59.275 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87331.log.gz 2026-03-31T21:27:59.275 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87332.log 2026-03-31T21:27:59.275 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87329.log.gz 2026-03-31T21:27:59.276 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87470.log 2026-03-31T21:27:59.276 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87332.log.gz 2026-03-31T21:27:59.276 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87474.log 2026-03-31T21:27:59.276 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87470.log.gz 2026-03-31T21:27:59.277 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87547.log 2026-03-31T21:27:59.277 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87474.log.gz 2026-03-31T21:27:59.277 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87546.log 2026-03-31T21:27:59.277 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87547.log.gz 2026-03-31T21:27:59.277 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87670.log 2026-03-31T21:27:59.278 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87546.log.gz 2026-03-31T21:27:59.278 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87669.log 2026-03-31T21:27:59.278 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87670.log.gz 2026-03-31T21:27:59.278 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87726.log 2026-03-31T21:27:59.278 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87669.log.gz 2026-03-31T21:27:59.279 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87725.log 2026-03-31T21:27:59.279 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87726.log.gz 2026-03-31T21:27:59.279 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87839.log 2026-03-31T21:27:59.279 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87725.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87725.log.gz 2026-03-31T21:27:59.280 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87893.log 2026-03-31T21:27:59.280 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87839.log.gz 2026-03-31T21:27:59.280 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87894.log 2026-03-31T21:27:59.280 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87893.log.gz 2026-03-31T21:27:59.280 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87980.log 2026-03-31T21:27:59.281 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87894.log.gz 2026-03-31T21:27:59.281 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88035.log 2026-03-31T21:27:59.281 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.87980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.87980.log.gz 2026-03-31T21:27:59.281 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88158.log 2026-03-31T21:27:59.282 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88035.log: 28.0% -- replaced with /var/log/ceph/ceph-client.admin.88035.log.gz 2026-03-31T21:27:59.282 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88159.log 2026-03-31T21:27:59.282 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88158.log.gz 2026-03-31T21:27:59.282 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88162.log 2026-03-31T21:27:59.282 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88159.log.gz 2026-03-31T21:27:59.283 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88165.log 2026-03-31T21:27:59.283 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88162.log.gz 2026-03-31T21:27:59.283 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88340.log 2026-03-31T21:27:59.283 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88165.log.gz 2026-03-31T21:27:59.284 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88361.log 2026-03-31T21:27:59.284 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88340.log.gz 2026-03-31T21:27:59.284 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88360.log 2026-03-31T21:27:59.284 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88361.log.gz 2026-03-31T21:27:59.284 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88359.log 2026-03-31T21:27:59.285 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88360.log.gz 2026-03-31T21:27:59.285 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88502.log 2026-03-31T21:27:59.285 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88359.log.gz 2026-03-31T21:27:59.285 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88503.log 2026-03-31T21:27:59.285 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88502.log.gz 2026-03-31T21:27:59.286 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88582.log 2026-03-31T21:27:59.286 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88503.log.gz 2026-03-31T21:27:59.286 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88592.log 2026-03-31T21:27:59.286 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88582.log.gz 2026-03-31T21:27:59.287 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88672.log 2026-03-31T21:27:59.287 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88592.log.gz 2026-03-31T21:27:59.287 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88726.log 2026-03-31T21:27:59.287 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88672.log.gz 2026-03-31T21:27:59.288 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88728.log 2026-03-31T21:27:59.288 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88726.log.gz 2026-03-31T21:27:59.288 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88776.log 2026-03-31T21:27:59.288 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88728.log.gz 2026-03-31T21:27:59.289 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88868.log 2026-03-31T21:27:59.289 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88776.log.gz 2026-03-31T21:27:59.289 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88994.log 2026-03-31T21:27:59.289 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88868.log.gz 2026-03-31T21:27:59.289 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88993.log 2026-03-31T21:27:59.290 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88994.log.gz 2026-03-31T21:27:59.290 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88992.log 2026-03-31T21:27:59.290 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88993.log.gz 2026-03-31T21:27:59.290 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88998.log 2026-03-31T21:27:59.290 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88992.log.gz 2026-03-31T21:27:59.291 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89124.log 2026-03-31T21:27:59.291 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.88998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.88998.log.gz 2026-03-31T21:27:59.291 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89191.log 2026-03-31T21:27:59.291 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89124.log.gz 2026-03-31T21:27:59.292 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89188.log 2026-03-31T21:27:59.292 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89191.log.gz 2026-03-31T21:27:59.292 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89194.log 2026-03-31T21:27:59.292 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89188.log.gz 2026-03-31T21:27:59.292 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89335.log 2026-03-31T21:27:59.293 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89194.log.gz 2026-03-31T21:27:59.293 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89336.log 2026-03-31T21:27:59.293 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89335.log.gz 2026-03-31T21:27:59.293 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89428.log 2026-03-31T21:27:59.293 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89336.log.gz 2026-03-31T21:27:59.294 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89407.log 2026-03-31T21:27:59.294 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89428.log.gz 2026-03-31T21:27:59.294 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89531.log 2026-03-31T21:27:59.294 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89407.log: 30.5% -- replaced with /var/log/ceph/ceph-client.admin.89407.log.gz 2026-03-31T21:27:59.295 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89532.log 2026-03-31T21:27:59.295 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89531.log.gz 2026-03-31T21:27:59.295 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89587.log 2026-03-31T21:27:59.295 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89532.log.gz 2026-03-31T21:27:59.296 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89652.log 2026-03-31T21:27:59.296 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89587.log.gz 2026-03-31T21:27:59.296 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89781.log 2026-03-31T21:27:59.296 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89652.log.gz 2026-03-31T21:27:59.296 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89779.log 2026-03-31T21:27:59.297 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89781.log.gz 2026-03-31T21:27:59.297 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89782.log 2026-03-31T21:27:59.297 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89779.log.gz 2026-03-31T21:27:59.297 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89780.log 2026-03-31T21:27:59.297 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89782.log.gz 2026-03-31T21:27:59.298 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89965.log 2026-03-31T21:27:59.298 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89780.log.gz 2026-03-31T21:27:59.298 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89978.log 2026-03-31T21:27:59.298 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89965.log.gz 2026-03-31T21:27:59.299 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89976.log 2026-03-31T21:27:59.299 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89978.log.gz 2026-03-31T21:27:59.299 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89974.log 2026-03-31T21:27:59.299 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89976.log.gz 2026-03-31T21:27:59.299 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90120.log 2026-03-31T21:27:59.300 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.89974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89974.log.gz 2026-03-31T21:27:59.300 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90119.log 2026-03-31T21:27:59.300 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90120.log.gz 2026-03-31T21:27:59.300 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90175.log 2026-03-31T21:27:59.300 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90119.log.gz 2026-03-31T21:27:59.301 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90202.log 2026-03-31T21:27:59.301 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90175.log.gz 2026-03-31T21:27:59.301 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90289.log 2026-03-31T21:27:59.301 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90202.log.gz 2026-03-31T21:27:59.302 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90326.log 2026-03-31T21:27:59.302 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90289.log.gz 2026-03-31T21:27:59.302 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90387.log 2026-03-31T21:27:59.302 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90326.log.gz 2026-03-31T21:27:59.303 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90397.log 2026-03-31T21:27:59.303 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90387.log.gz 2026-03-31T21:27:59.303 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90485.log 2026-03-31T21:27:59.303 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90397.log.gz 2026-03-31T21:27:59.304 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90612.log 2026-03-31T21:27:59.304 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90485.log.gz 2026-03-31T21:27:59.304 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90615.log 2026-03-31T21:27:59.304 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90612.log.gz 2026-03-31T21:27:59.304 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90614.log 2026-03-31T21:27:59.305 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90615.log.gz 2026-03-31T21:27:59.305 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90613.log 2026-03-31T21:27:59.305 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90614.log.gz 2026-03-31T21:27:59.305 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90782.log 2026-03-31T21:27:59.305 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90613.log.gz 2026-03-31T21:27:59.306 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90781.log 2026-03-31T21:27:59.306 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90782.log.gz 2026-03-31T21:27:59.306 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90784.log 2026-03-31T21:27:59.306 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90781.log.gz 2026-03-31T21:27:59.307 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90871.log 2026-03-31T21:27:59.307 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90784.log.gz 2026-03-31T21:27:59.307 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90926.log 2026-03-31T21:27:59.307 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90871.log.gz 2026-03-31T21:27:59.307 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90980.log 2026-03-31T21:27:59.308 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90926.log.gz 2026-03-31T21:27:59.308 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90979.log 2026-03-31T21:27:59.308 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90980.log.gz 2026-03-31T21:27:59.308 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91036.log 2026-03-31T21:27:59.308 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.90979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90979.log.gz 2026-03-31T21:27:59.309 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91122.log 2026-03-31T21:27:59.309 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91036.log.gz 2026-03-31T21:27:59.309 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91178.log 2026-03-31T21:27:59.309 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91122.log.gz 2026-03-31T21:27:59.310 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91173.log 2026-03-31T21:27:59.310 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91178.log.gz 2026-03-31T21:27:59.310 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91269.log 2026-03-31T21:27:59.310 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91173.log.gz 2026-03-31T21:27:59.310 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91393.log 2026-03-31T21:27:59.311 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91269.log.gz 2026-03-31T21:27:59.311 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91398.log 2026-03-31T21:27:59.311 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91393.log.gz 2026-03-31T21:27:59.311 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91394.log 2026-03-31T21:27:59.311 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91398.log.gz 2026-03-31T21:27:59.312 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91399.log 2026-03-31T21:27:59.312 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91394.log.gz 2026-03-31T21:27:59.312 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91593.log 2026-03-31T21:27:59.312 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91399.log.gz 2026-03-31T21:27:59.313 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91592.log 2026-03-31T21:27:59.313 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91593.log.gz 2026-03-31T21:27:59.313 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91594.log 2026-03-31T21:27:59.313 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91592.log.gz 2026-03-31T21:27:59.313 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91595.log 2026-03-31T21:27:59.314 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91594.log.gz 2026-03-31T21:27:59.314 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91710.log 2026-03-31T21:27:59.314 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91595.log.gz 2026-03-31T21:27:59.314 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91791.log 2026-03-31T21:27:59.315 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91710.log.gz 2026-03-31T21:27:59.315 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91793.log 2026-03-31T21:27:59.315 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91791.log.gz 2026-03-31T21:27:59.315 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91792.log 2026-03-31T21:27:59.315 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91793.log.gz 2026-03-31T21:27:59.316 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91906.log 2026-03-31T21:27:59.316 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91792.log.gz 2026-03-31T21:27:59.316 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92031.log 2026-03-31T21:27:59.316 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.91906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91906.log.gz 2026-03-31T21:27:59.317 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92033.log 2026-03-31T21:27:59.317 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92031.log.gz 2026-03-31T21:27:59.317 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92034.log 2026-03-31T21:27:59.317 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92033.log.gz 2026-03-31T21:27:59.318 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92036.log 2026-03-31T21:27:59.318 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92034.log.gz 2026-03-31T21:27:59.318 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92216.log 2026-03-31T21:27:59.318 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92036.log.gz 2026-03-31T21:27:59.319 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92227.log 2026-03-31T21:27:59.319 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92216.log.gz 2026-03-31T21:27:59.319 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92232.log 2026-03-31T21:27:59.319 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92227.log.gz 2026-03-31T21:27:59.319 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92231.log 2026-03-31T21:27:59.319 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92232.log.gz 2026-03-31T21:27:59.320 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92347.log 2026-03-31T21:27:59.320 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92231.log.gz 2026-03-31T21:27:59.320 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92475.log 2026-03-31T21:27:59.320 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92347.log.gz 2026-03-31T21:27:59.321 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92477.log 2026-03-31T21:27:59.321 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92475.log.gz 2026-03-31T21:27:59.321 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92474.log 2026-03-31T21:27:59.321 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92477.log.gz 2026-03-31T21:27:59.322 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92476.log 2026-03-31T21:27:59.322 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92474.log.gz 2026-03-31T21:27:59.322 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92592.log 2026-03-31T21:27:59.322 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92476.log.gz 2026-03-31T21:27:59.322 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92663.log 2026-03-31T21:27:59.323 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92592.log.gz 2026-03-31T21:27:59.323 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92666.log 2026-03-31T21:27:59.323 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92663.log.gz 2026-03-31T21:27:59.323 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92701.log 2026-03-31T21:27:59.323 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92666.log.gz 2026-03-31T21:27:59.324 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92788.log 2026-03-31T21:27:59.324 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92701.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92701.log.gz 2026-03-31T21:27:59.324 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92851.log 2026-03-31T21:27:59.324 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92788.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92788.log.gz 2026-03-31T21:27:59.325 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92861.log 2026-03-31T21:27:59.325 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92851.log.gz 2026-03-31T21:27:59.325 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92909.log 2026-03-31T21:27:59.325 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92861.log.gz 2026-03-31T21:27:59.325 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92984.log 2026-03-31T21:27:59.326 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92909.log.gz 2026-03-31T21:27:59.326 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93112.log 2026-03-31T21:27:59.326 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.92984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.92984.log.gz 2026-03-31T21:27:59.326 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93114.log 2026-03-31T21:27:59.326 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93112.log.gz 2026-03-31T21:27:59.327 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93115.log 2026-03-31T21:27:59.327 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93114.log.gz 2026-03-31T21:27:59.327 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93113.log 2026-03-31T21:27:59.327 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93115.log.gz 2026-03-31T21:27:59.328 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93282.log 2026-03-31T21:27:59.328 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93113.log.gz 2026-03-31T21:27:59.328 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93307.log 2026-03-31T21:27:59.328 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93282.log.gz 2026-03-31T21:27:59.329 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93308.log 2026-03-31T21:27:59.329 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93307.log.gz 2026-03-31T21:27:59.329 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93311.log 2026-03-31T21:27:59.329 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93308.log.gz 2026-03-31T21:27:59.329 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93426.log 2026-03-31T21:27:59.330 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93311.log.gz 2026-03-31T21:27:59.330 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93516.log 2026-03-31T21:27:59.330 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93426.log.gz 2026-03-31T21:27:59.330 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93517.log 2026-03-31T21:27:59.330 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93516.log.gz 2026-03-31T21:27:59.331 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93506.log 2026-03-31T21:27:59.331 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93517.log.gz 2026-03-31T21:27:59.331 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93622.log 2026-03-31T21:27:59.331 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93506.log.gz 2026-03-31T21:27:59.332 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93676.log 2026-03-31T21:27:59.332 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93622.log.gz 2026-03-31T21:27:59.332 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93677.log 2026-03-31T21:27:59.332 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93676.log.gz 2026-03-31T21:27:59.333 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93740.log 2026-03-31T21:27:59.333 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93677.log.gz 2026-03-31T21:27:59.333 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93818.log 2026-03-31T21:27:59.333 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93740.log.gz 2026-03-31T21:27:59.334 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93945.log 2026-03-31T21:27:59.334 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93818.log.gz 2026-03-31T21:27:59.334 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93948.log 2026-03-31T21:27:59.334 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93945.log.gz 2026-03-31T21:27:59.334 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93946.log 2026-03-31T21:27:59.334 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93948.log.gz 2026-03-31T21:27:59.335 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93947.log 2026-03-31T21:27:59.335 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93946.log.gz 2026-03-31T21:27:59.335 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94116.log 2026-03-31T21:27:59.335 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.93947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93947.log.gz 2026-03-31T21:27:59.336 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94117.log 2026-03-31T21:27:59.336 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94116.log.gz 2026-03-31T21:27:59.336 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94115.log 2026-03-31T21:27:59.336 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94117.log.gz 2026-03-31T21:27:59.337 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94200.log 2026-03-31T21:27:59.337 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94115.log.gz 2026-03-31T21:27:59.337 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94259.log 2026-03-31T21:27:59.337 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94200.log.gz 2026-03-31T21:27:59.337 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94322.log 2026-03-31T21:27:59.338 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94259.log.gz 2026-03-31T21:27:59.338 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94323.log 2026-03-31T21:27:59.338 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94322.log.gz 2026-03-31T21:27:59.338 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94376.log 2026-03-31T21:27:59.338 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94323.log.gz 2026-03-31T21:27:59.339 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94455.log 2026-03-31T21:27:59.339 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94376.log.gz 2026-03-31T21:27:59.339 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94577.log 2026-03-31T21:27:59.339 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94455.log.gz 2026-03-31T21:27:59.340 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94584.log 2026-03-31T21:27:59.340 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94577.log.gz 2026-03-31T21:27:59.340 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94583.log 2026-03-31T21:27:59.340 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94584.log.gz 2026-03-31T21:27:59.340 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94585.log 2026-03-31T21:27:59.341 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94583.log.gz 2026-03-31T21:27:59.341 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94753.log 2026-03-31T21:27:59.341 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94585.log.gz 2026-03-31T21:27:59.341 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94754.log 2026-03-31T21:27:59.341 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94753.log.gz 2026-03-31T21:27:59.342 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94752.log 2026-03-31T21:27:59.342 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94754.log.gz 2026-03-31T21:27:59.342 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94803.log 2026-03-31T21:27:59.342 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94752.log.gz 2026-03-31T21:27:59.343 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94896.log 2026-03-31T21:27:59.343 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94803.log.gz 2026-03-31T21:27:59.343 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94952.log 2026-03-31T21:27:59.343 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94896.log.gz 2026-03-31T21:27:59.344 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94950.log 2026-03-31T21:27:59.344 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94952.log.gz 2026-03-31T21:27:59.344 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94998.log 2026-03-31T21:27:59.344 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94950.log.gz 2026-03-31T21:27:59.344 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95092.log 2026-03-31T21:27:59.344 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.94998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94998.log.gz 2026-03-31T21:27:59.345 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95213.log 2026-03-31T21:27:59.345 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95092.log.gz 2026-03-31T21:27:59.345 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95220.log 2026-03-31T21:27:59.345 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95213.log.gz 2026-03-31T21:27:59.346 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95218.log 2026-03-31T21:27:59.346 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95220.log.gz 2026-03-31T21:27:59.352 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95222.log 2026-03-31T21:27:59.353 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95218.log.gz 2026-03-31T21:27:59.369 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95400.log 2026-03-31T21:27:59.369 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95222.log.gz 2026-03-31T21:27:59.385 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95416.log 2026-03-31T21:27:59.385 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95400.log.gz 2026-03-31T21:27:59.399 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95418.log 2026-03-31T21:27:59.399 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95416.log.gz 2026-03-31T21:27:59.413 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95417.log 2026-03-31T21:27:59.414 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95418.log.gz 2026-03-31T21:27:59.427 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95560.log 2026-03-31T21:27:59.427 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95417.log.gz 2026-03-31T21:27:59.434 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95559.log 2026-03-31T21:27:59.435 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95560.log.gz 2026-03-31T21:27:59.441 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95642.log 2026-03-31T21:27:59.442 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95559.log.gz 2026-03-31T21:27:59.452 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95644.log 2026-03-31T21:27:59.453 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95642.log.gz 2026-03-31T21:27:59.459 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95729.log 2026-03-31T21:27:59.459 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95644.log.gz 2026-03-31T21:27:59.467 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95757.log 2026-03-31T21:27:59.467 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95729.log.gz 2026-03-31T21:27:59.474 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95830.log 2026-03-31T21:27:59.475 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95757.log.gz 2026-03-31T21:27:59.482 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95829.log 2026-03-31T21:27:59.483 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95830.log.gz 2026-03-31T21:27:59.490 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95925.log 2026-03-31T21:27:59.491 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95829.log.gz 2026-03-31T21:27:59.498 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96054.log 2026-03-31T21:27:59.499 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.95925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.95925.log.gz 2026-03-31T21:27:59.505 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96049.log 2026-03-31T21:27:59.506 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96054.log.gz 2026-03-31T21:27:59.512 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96056.log 2026-03-31T21:27:59.513 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96049.log.gz 2026-03-31T21:27:59.522 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96055.log 2026-03-31T21:27:59.523 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96056.log.gz 2026-03-31T21:27:59.529 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96240.log 2026-03-31T21:27:59.530 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96055.log.gz 2026-03-31T21:27:59.538 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96248.log 2026-03-31T21:27:59.539 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96240.log.gz 2026-03-31T21:27:59.545 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96249.log 2026-03-31T21:27:59.546 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96248.log.gz 2026-03-31T21:27:59.552 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96252.log 2026-03-31T21:27:59.553 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96249.log.gz 2026-03-31T21:27:59.562 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96367.log 2026-03-31T21:27:59.563 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96252.log.gz 2026-03-31T21:27:59.569 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96447.log 2026-03-31T21:27:59.569 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96367.log.gz 2026-03-31T21:27:59.578 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96449.log 2026-03-31T21:27:59.578 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96447.log.gz 2026-03-31T21:27:59.585 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96458.log 2026-03-31T21:27:59.585 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96449.log.gz 2026-03-31T21:27:59.595 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96563.log 2026-03-31T21:27:59.596 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96458.log.gz 2026-03-31T21:27:59.600 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96626.log 2026-03-31T21:27:59.601 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96563.log.gz 2026-03-31T21:27:59.607 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96627.log 2026-03-31T21:27:59.607 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96626.log.gz 2026-03-31T21:27:59.616 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96666.log 2026-03-31T21:27:59.616 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96627.log.gz 2026-03-31T21:27:59.626 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96759.log 2026-03-31T21:27:59.626 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96666.log.gz 2026-03-31T21:27:59.633 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96884.log 2026-03-31T21:27:59.634 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96759.log.gz 2026-03-31T21:27:59.642 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96889.log 2026-03-31T21:27:59.643 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96884.log.gz 2026-03-31T21:27:59.650 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96886.log 2026-03-31T21:27:59.651 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96889.log.gz 2026-03-31T21:27:59.666 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96888.log 2026-03-31T21:27:59.667 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96886.log.gz 2026-03-31T21:27:59.673 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97029.log 2026-03-31T21:27:59.674 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.96888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96888.log.gz 2026-03-31T21:27:59.681 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97028.log 2026-03-31T21:27:59.682 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97029.log.gz 2026-03-31T21:27:59.690 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97058.log 2026-03-31T21:27:59.691 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97028.log.gz 2026-03-31T21:27:59.697 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97149.log 2026-03-31T21:27:59.698 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97058.log.gz 2026-03-31T21:27:59.704 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97200.log 2026-03-31T21:27:59.705 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97149.log.gz 2026-03-31T21:27:59.713 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97228.log 2026-03-31T21:27:59.714 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97200.log.gz 2026-03-31T21:27:59.720 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97275.log 2026-03-31T21:27:59.721 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97228.log.gz 2026-03-31T21:27:59.728 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97343.log 2026-03-31T21:27:59.728 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97275.log.gz 2026-03-31T21:27:59.737 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97396.log 2026-03-31T21:27:59.738 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97343.log.gz 2026-03-31T21:27:59.744 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97445.log 2026-03-31T21:27:59.745 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97396.log.gz 2026-03-31T21:27:59.753 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97574.log 2026-03-31T21:27:59.754 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97445.log.gz 2026-03-31T21:27:59.760 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97575.log 2026-03-31T21:27:59.760 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97574.log.gz 2026-03-31T21:27:59.769 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97573.log 2026-03-31T21:27:59.770 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97575.log.gz 2026-03-31T21:27:59.776 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97571.log 2026-03-31T21:27:59.776 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97573.log.gz 2026-03-31T21:27:59.785 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97678.log 2026-03-31T21:27:59.785 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97571.log: 30.9% -- replaced with /var/log/ceph/ceph-client.admin.97571.log.gz 2026-03-31T21:27:59.792 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97733.log 2026-03-31T21:27:59.792 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97678.log.gz 2026-03-31T21:27:59.799 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97742.log 2026-03-31T21:27:59.799 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97733.log.gz 2026-03-31T21:27:59.811 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97823.log 2026-03-31T21:27:59.811 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97742.log: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.97742.log.gz 2026-03-31T21:27:59.818 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97886.log 2026-03-31T21:27:59.818 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97823.log.gz 2026-03-31T21:27:59.835 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97931.log 2026-03-31T21:27:59.835 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97886.log.gz 2026-03-31T21:27:59.841 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97982.log 2026-03-31T21:27:59.842 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97931.log.gz 2026-03-31T21:27:59.851 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98031.log 2026-03-31T21:27:59.852 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.97982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.97982.log.gz 2026-03-31T21:27:59.857 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98082.log 2026-03-31T21:27:59.858 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98031.log.gz 2026-03-31T21:27:59.864 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98111.log 2026-03-31T21:27:59.864 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98082.log.gz 2026-03-31T21:27:59.872 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98180.log 2026-03-31T21:27:59.873 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98111.log.gz 2026-03-31T21:27:59.880 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98310.log 2026-03-31T21:27:59.880 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98180.log.gz 2026-03-31T21:27:59.887 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98308.log 2026-03-31T21:27:59.887 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98310.log.gz 2026-03-31T21:27:59.895 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98309.log 2026-03-31T21:27:59.895 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98308.log.gz 2026-03-31T21:27:59.903 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98307.log 2026-03-31T21:27:59.904 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98309.log.gz 2026-03-31T21:27:59.910 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98477.log 2026-03-31T21:27:59.910 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98307.log.gz 2026-03-31T21:27:59.919 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98479.log 2026-03-31T21:27:59.919 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98477.log.gz 2026-03-31T21:27:59.928 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98478.log 2026-03-31T21:27:59.928 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98479.log.gz 2026-03-31T21:27:59.935 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98551.log 2026-03-31T21:27:59.935 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98478.log.gz 2026-03-31T21:27:59.944 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98621.log 2026-03-31T21:27:59.944 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98551.log.gz 2026-03-31T21:27:59.952 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98675.log 2026-03-31T21:27:59.952 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98621.log.gz 2026-03-31T21:27:59.960 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98674.log 2026-03-31T21:27:59.960 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98675.log.gz 2026-03-31T21:27:59.969 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98727.log 2026-03-31T21:27:59.970 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98674.log.gz 2026-03-31T21:27:59.976 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98817.log 2026-03-31T21:27:59.977 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98727.log.gz 2026-03-31T21:27:59.983 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98866.log 2026-03-31T21:27:59.983 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98817.log.gz 2026-03-31T21:27:59.990 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98991.log 2026-03-31T21:27:59.990 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98866.log.gz 2026-03-31T21:27:59.998 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98992.log 2026-03-31T21:27:59.998 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98991.log.gz 2026-03-31T21:28:00.006 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98996.log 2026-03-31T21:28:00.006 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98992.log.gz 2026-03-31T21:28:00.015 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98994.log 2026-03-31T21:28:00.015 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98996.log.gz 2026-03-31T21:28:00.022 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99167.log 2026-03-31T21:28:00.022 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.98994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98994.log.gz 2026-03-31T21:28:00.031 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99189.log 2026-03-31T21:28:00.031 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99167.log.gz 2026-03-31T21:28:00.040 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99191.log 2026-03-31T21:28:00.041 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99189.log.gz 2026-03-31T21:28:00.047 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99192.log 2026-03-31T21:28:00.047 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99191.log.gz 2026-03-31T21:28:00.058 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99307.log 2026-03-31T21:28:00.058 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99192.log.gz 2026-03-31T21:28:00.065 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99388.log 2026-03-31T21:28:00.066 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99307.log.gz 2026-03-31T21:28:00.072 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99389.log 2026-03-31T21:28:00.073 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99388.log.gz 2026-03-31T21:28:00.081 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99387.log 2026-03-31T21:28:00.081 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99389.log.gz 2026-03-31T21:28:00.088 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99503.log 2026-03-31T21:28:00.088 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99387.log.gz 2026-03-31T21:28:00.098 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99557.log 2026-03-31T21:28:00.099 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99503.log.gz 2026-03-31T21:28:00.103 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99556.log 2026-03-31T21:28:00.103 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99557.log.gz 2026-03-31T21:28:00.110 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99629.log 2026-03-31T21:28:00.110 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99556.log.gz 2026-03-31T21:28:00.118 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99699.log 2026-03-31T21:28:00.118 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99629.log.gz 2026-03-31T21:28:00.127 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99727.log 2026-03-31T21:28:00.127 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99699.log.gz 2026-03-31T21:28:00.134 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99773.log 2026-03-31T21:28:00.134 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99727.log.gz 2026-03-31T21:28:00.146 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99839.log 2026-03-31T21:28:00.146 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99773.log.gz 2026-03-31T21:28:00.153 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99895.log 2026-03-31T21:28:00.154 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99839.log.gz 2026-03-31T21:28:00.162 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100023.log 2026-03-31T21:28:00.162 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.99895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.99895.log.gz 2026-03-31T21:28:00.170 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100022.log 2026-03-31T21:28:00.171 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100023.log.gz 2026-03-31T21:28:00.177 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100025.log 2026-03-31T21:28:00.177 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100022.log.gz 2026-03-31T21:28:00.184 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100024.log 2026-03-31T21:28:00.184 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100025.log.gz 2026-03-31T21:28:00.191 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100167.log 2026-03-31T21:28:00.192 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100024.log.gz 2026-03-31T21:28:00.199 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100166.log 2026-03-31T21:28:00.199 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100167.log.gz 2026-03-31T21:28:00.206 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100250.log 2026-03-31T21:28:00.206 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100166.log.gz 2026-03-31T21:28:00.214 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100248.log 2026-03-31T21:28:00.214 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100250.log.gz 2026-03-31T21:28:00.221 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100363.log 2026-03-31T21:28:00.221 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100248.log.gz 2026-03-31T21:28:00.232 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100362.log 2026-03-31T21:28:00.233 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100363.log.gz 2026-03-31T21:28:00.239 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100416.log 2026-03-31T21:28:00.240 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100362.log.gz 2026-03-31T21:28:00.251 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100417.log 2026-03-31T21:28:00.252 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100416.log.gz 2026-03-31T21:28:00.258 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100532.log 2026-03-31T21:28:00.258 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100417.log.gz 2026-03-31T21:28:00.267 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100613.log 2026-03-31T21:28:00.267 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100532.log.gz 2026-03-31T21:28:00.274 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100611.log 2026-03-31T21:28:00.274 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100613.log.gz 2026-03-31T21:28:00.282 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100623.log 2026-03-31T21:28:00.282 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100611.log.gz 2026-03-31T21:28:00.290 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100728.log 2026-03-31T21:28:00.290 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100623.log.gz 2026-03-31T21:28:00.299 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100856.log 2026-03-31T21:28:00.299 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100728.log.gz 2026-03-31T21:28:00.309 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100858.log 2026-03-31T21:28:00.310 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100856.log.gz 2026-03-31T21:28:00.318 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100855.log 2026-03-31T21:28:00.318 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100858.log.gz 2026-03-31T21:28:00.326 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100857.log 2026-03-31T21:28:00.326 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100855.log.gz 2026-03-31T21:28:00.333 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100995.log 2026-03-31T21:28:00.333 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100857.log.gz 2026-03-31T21:28:00.341 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100998.log 2026-03-31T21:28:00.341 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100995.log.gz 2026-03-31T21:28:00.355 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101027.log 2026-03-31T21:28:00.356 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.100998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.100998.log.gz 2026-03-31T21:28:00.356 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101118.log 2026-03-31T21:28:00.356 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101027.log.gz 2026-03-31T21:28:00.357 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101169.log 2026-03-31T21:28:00.357 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101118.log.gz 2026-03-31T21:28:00.357 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101222.log 2026-03-31T21:28:00.358 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101169.log.gz 2026-03-31T21:28:00.358 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101224.log 2026-03-31T21:28:00.358 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101222.log.gz 2026-03-31T21:28:00.359 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101312.log 2026-03-31T21:28:00.359 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101224.log.gz 2026-03-31T21:28:00.359 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101365.log 2026-03-31T21:28:00.360 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101312.log.gz 2026-03-31T21:28:00.360 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101426.log 2026-03-31T21:28:00.360 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101365.log.gz 2026-03-31T21:28:00.361 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101420.log 2026-03-31T21:28:00.361 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101426.log.gz 2026-03-31T21:28:00.362 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101508.log 2026-03-31T21:28:00.362 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101420.log.gz 2026-03-31T21:28:00.362 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101561.log 2026-03-31T21:28:00.363 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101508.log.gz 2026-03-31T21:28:00.363 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101691.log 2026-03-31T21:28:00.364 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101561.log.gz 2026-03-31T21:28:00.364 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101689.log 2026-03-31T21:28:00.364 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101691.log.gz 2026-03-31T21:28:00.365 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101690.log 2026-03-31T21:28:00.365 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101689.log.gz 2026-03-31T21:28:00.365 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101685.log 2026-03-31T21:28:00.366 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101690.log.gz 2026-03-31T21:28:00.366 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101873.log 2026-03-31T21:28:00.366 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101685.log.gz 2026-03-31T21:28:00.367 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101884.log 2026-03-31T21:28:00.367 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101873.log.gz 2026-03-31T21:28:00.367 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101886.log 2026-03-31T21:28:00.368 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101884.log.gz 2026-03-31T21:28:00.368 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101887.log 2026-03-31T21:28:00.368 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101886.log.gz 2026-03-31T21:28:00.369 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102029.log 2026-03-31T21:28:00.369 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.101887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.101887.log.gz 2026-03-31T21:28:00.369 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102028.log 2026-03-31T21:28:00.370 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102029.log.gz 2026-03-31T21:28:00.370 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102102.log 2026-03-31T21:28:00.370 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102028.log.gz 2026-03-31T21:28:00.371 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102103.log 2026-03-31T21:28:00.371 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102102.log: 30.0% -- replaced with /var/log/ceph/ceph-client.admin.102102.log.gz 2026-03-31T21:28:00.371 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102198.log 2026-03-31T21:28:00.372 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102103.log.gz 2026-03-31T21:28:00.372 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102259.log 2026-03-31T21:28:00.372 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102198.log.gz 2026-03-31T21:28:00.373 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102251.log 2026-03-31T21:28:00.373 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102259.log.gz 2026-03-31T21:28:00.373 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102345.log 2026-03-31T21:28:00.374 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102251.log.gz 2026-03-31T21:28:00.374 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102471.log 2026-03-31T21:28:00.374 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102345.log.gz 2026-03-31T21:28:00.375 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102473.log 2026-03-31T21:28:00.375 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102471.log.gz 2026-03-31T21:28:00.375 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102469.log 2026-03-31T21:28:00.376 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102473.log.gz 2026-03-31T21:28:00.376 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102475.log 2026-03-31T21:28:00.376 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102469.log.gz 2026-03-31T21:28:00.377 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102636.log 2026-03-31T21:28:00.377 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102475.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102475.log.gz 2026-03-31T21:28:00.377 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102642.log 2026-03-31T21:28:00.378 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102636.log.gz 2026-03-31T21:28:00.378 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102641.log 2026-03-31T21:28:00.379 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102642.log.gz 2026-03-31T21:28:00.379 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102737.log 2026-03-31T21:28:00.379 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102641.log.gz 2026-03-31T21:28:00.380 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102786.log 2026-03-31T21:28:00.380 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102737.log.gz 2026-03-31T21:28:00.380 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102840.log 2026-03-31T21:28:00.381 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102786.log.gz 2026-03-31T21:28:00.381 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102842.log 2026-03-31T21:28:00.381 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102840.log.gz 2026-03-31T21:28:00.382 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102929.log 2026-03-31T21:28:00.382 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102842.log.gz 2026-03-31T21:28:00.382 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102982.log 2026-03-31T21:28:00.383 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102929.log.gz 2026-03-31T21:28:00.383 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103037.log 2026-03-31T21:28:00.383 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.102982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.102982.log.gz 2026-03-31T21:28:00.384 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103036.log 2026-03-31T21:28:00.384 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103037.log.gz 2026-03-31T21:28:00.384 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103129.log 2026-03-31T21:28:00.385 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103036.log.gz 2026-03-31T21:28:00.385 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103259.log 2026-03-31T21:28:00.385 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103129.log.gz 2026-03-31T21:28:00.386 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103257.log 2026-03-31T21:28:00.386 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103259.log.gz 2026-03-31T21:28:00.386 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103256.log 2026-03-31T21:28:00.387 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103257.log.gz 2026-03-31T21:28:00.387 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103258.log 2026-03-31T21:28:00.387 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103256.log.gz 2026-03-31T21:28:00.388 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103438.log 2026-03-31T21:28:00.388 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103258.log.gz 2026-03-31T21:28:00.388 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103450.log 2026-03-31T21:28:00.389 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103438.log.gz 2026-03-31T21:28:00.389 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103453.log 2026-03-31T21:28:00.389 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103450.log.gz 2026-03-31T21:28:00.390 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103455.log 2026-03-31T21:28:00.390 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103453.log.gz 2026-03-31T21:28:00.390 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103597.log 2026-03-31T21:28:00.391 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103455.log.gz 2026-03-31T21:28:00.391 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103596.log 2026-03-31T21:28:00.391 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103597.log.gz 2026-03-31T21:28:00.392 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103679.log 2026-03-31T21:28:00.392 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103596.log.gz 2026-03-31T21:28:00.392 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103663.log 2026-03-31T21:28:00.393 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103679.log.gz 2026-03-31T21:28:00.393 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103766.log 2026-03-31T21:28:00.393 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103663.log.gz 2026-03-31T21:28:00.394 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103821.log 2026-03-31T21:28:00.394 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103766.log.gz 2026-03-31T21:28:00.394 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103822.log 2026-03-31T21:28:00.395 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103821.log.gz 2026-03-31T21:28:00.395 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103884.log 2026-03-31T21:28:00.395 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103822.log.gz 2026-03-31T21:28:00.396 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103962.log 2026-03-31T21:28:00.396 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103884.log.gz 2026-03-31T21:28:00.396 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104090.log 2026-03-31T21:28:00.397 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.103962.log: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.103962.log.gz 2026-03-31T21:28:00.397 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104088.log 2026-03-31T21:28:00.397 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104090.log.gz 2026-03-31T21:28:00.398 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104091.log 2026-03-31T21:28:00.398 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104088.log.gz 2026-03-31T21:28:00.398 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104092.log 2026-03-31T21:28:00.399 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104091.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104091.log.gz 2026-03-31T21:28:00.399 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104259.log 2026-03-31T21:28:00.399 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104092.log.gz 2026-03-31T21:28:00.400 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104285.log 2026-03-31T21:28:00.400 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104259.log.gz 2026-03-31T21:28:00.400 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104287.log 2026-03-31T21:28:00.401 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104285.log.gz 2026-03-31T21:28:00.401 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104288.log 2026-03-31T21:28:00.402 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104287.log.gz 2026-03-31T21:28:00.402 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104403.log 2026-03-31T21:28:00.402 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104288.log.gz 2026-03-31T21:28:00.403 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104483.log 2026-03-31T21:28:00.403 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104403.log.gz 2026-03-31T21:28:00.403 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104482.log 2026-03-31T21:28:00.404 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104483.log.gz 2026-03-31T21:28:00.404 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104494.log 2026-03-31T21:28:00.404 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104482.log.gz 2026-03-31T21:28:00.405 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104599.log 2026-03-31T21:28:00.405 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104494.log.gz 2026-03-31T21:28:00.405 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104728.log 2026-03-31T21:28:00.406 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104599.log.gz 2026-03-31T21:28:00.406 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104729.log 2026-03-31T21:28:00.406 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104728.log.gz 2026-03-31T21:28:00.407 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104727.log 2026-03-31T21:28:00.407 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104729.log.gz 2026-03-31T21:28:00.407 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104726.log 2026-03-31T21:28:00.408 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104727.log.gz 2026-03-31T21:28:00.408 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104897.log 2026-03-31T21:28:00.408 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104726.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104726.log.gz 2026-03-31T21:28:00.408 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104896.log 2026-03-31T21:28:00.409 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104897.log.gz 2026-03-31T21:28:00.409 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104898.log 2026-03-31T21:28:00.410 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104896.log.gz 2026-03-31T21:28:00.410 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104985.log 2026-03-31T21:28:00.410 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104898.log.gz 2026-03-31T21:28:00.410 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105040.log 2026-03-31T21:28:00.411 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.104985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104985.log.gz 2026-03-31T21:28:00.411 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105095.log 2026-03-31T21:28:00.412 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105040.log.gz 2026-03-31T21:28:00.412 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105094.log 2026-03-31T21:28:00.412 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105095.log.gz 2026-03-31T21:28:00.413 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105171.log 2026-03-31T21:28:00.413 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105094.log.gz 2026-03-31T21:28:00.413 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105236.log 2026-03-31T21:28:00.414 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105171.log.gz 2026-03-31T21:28:00.414 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105366.log 2026-03-31T21:28:00.414 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105236.log.gz 2026-03-31T21:28:00.414 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105364.log 2026-03-31T21:28:00.415 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105366.log.gz 2026-03-31T21:28:00.415 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105365.log 2026-03-31T21:28:00.415 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105364.log: 29.8% -- replaced with /var/log/ceph/ceph-client.admin.105364.log.gz 2026-03-31T21:28:00.416 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105363.log 2026-03-31T21:28:00.416 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105365.log.gz 2026-03-31T21:28:00.416 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105504.log 2026-03-31T21:28:00.417 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105363.log.gz 2026-03-31T21:28:00.417 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105503.log 2026-03-31T21:28:00.417 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105504.log.gz 2026-03-31T21:28:00.418 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105533.log 2026-03-31T21:28:00.418 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105503.log.gz 2026-03-31T21:28:00.418 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105626.log 2026-03-31T21:28:00.419 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105533.log.gz 2026-03-31T21:28:00.419 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105677.log 2026-03-31T21:28:00.419 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105626.log.gz 2026-03-31T21:28:00.420 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105705.log 2026-03-31T21:28:00.420 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105677.log.gz 2026-03-31T21:28:00.420 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105753.log 2026-03-31T21:28:00.421 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105705.log.gz 2026-03-31T21:28:00.421 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105820.log 2026-03-31T21:28:00.421 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105753.log.gz 2026-03-31T21:28:00.422 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105873.log 2026-03-31T21:28:00.422 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105820.log: 29.4% -- replaced with /var/log/ceph/ceph-client.admin.105820.log.gz 2026-03-31T21:28:00.422 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105920.log 2026-03-31T21:28:00.423 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105873.log.gz 2026-03-31T21:28:00.423 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105948.log 2026-03-31T21:28:00.423 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105920.log.gz 2026-03-31T21:28:00.424 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106020.log 2026-03-31T21:28:00.424 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.105948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105948.log.gz 2026-03-31T21:28:00.425 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106149.log 2026-03-31T21:28:00.425 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106020.log.gz 2026-03-31T21:28:00.425 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106144.log 2026-03-31T21:28:00.426 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106149.log.gz 2026-03-31T21:28:00.426 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106148.log 2026-03-31T21:28:00.426 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106144.log.gz 2026-03-31T21:28:00.426 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106150.log 2026-03-31T21:28:00.427 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106148.log.gz 2026-03-31T21:28:00.427 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106313.log 2026-03-31T21:28:00.428 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106150.log.gz 2026-03-31T21:28:00.428 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106319.log 2026-03-31T21:28:00.428 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106313.log.gz 2026-03-31T21:28:00.428 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106316.log 2026-03-31T21:28:00.429 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106319.log.gz 2026-03-31T21:28:00.429 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106386.log 2026-03-31T21:28:00.429 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106316.log.gz 2026-03-31T21:28:00.430 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106461.log 2026-03-31T21:28:00.430 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106386.log.gz 2026-03-31T21:28:00.430 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106515.log 2026-03-31T21:28:00.431 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106461.log.gz 2026-03-31T21:28:00.431 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106516.log 2026-03-31T21:28:00.431 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106515.log.gz 2026-03-31T21:28:00.432 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106564.log 2026-03-31T21:28:00.432 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106516.log.gz 2026-03-31T21:28:00.432 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106657.log 2026-03-31T21:28:00.433 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106564.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106564.log.gz 2026-03-31T21:28:00.433 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106694.log 2026-03-31T21:28:00.433 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106657.log.gz 2026-03-31T21:28:00.434 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106755.log 2026-03-31T21:28:00.434 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106694.log.gz 2026-03-31T21:28:00.434 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106883.log 2026-03-31T21:28:00.435 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106755.log.gz 2026-03-31T21:28:00.435 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106885.log 2026-03-31T21:28:00.435 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106883.log.gz 2026-03-31T21:28:00.436 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106884.log 2026-03-31T21:28:00.436 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106885.log.gz 2026-03-31T21:28:00.436 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106882.log 2026-03-31T21:28:00.437 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106884.log.gz 2026-03-31T21:28:00.437 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107000.log 2026-03-31T21:28:00.437 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.106882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106882.log.gz 2026-03-31T21:28:00.438 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107074.log 2026-03-31T21:28:00.438 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107000.log.gz 2026-03-31T21:28:00.438 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107072.log 2026-03-31T21:28:00.439 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107074.log.gz 2026-03-31T21:28:00.439 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107131.log 2026-03-31T21:28:00.439 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107072.log.gz 2026-03-31T21:28:00.440 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107196.log 2026-03-31T21:28:00.440 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107131.log.gz 2026-03-31T21:28:00.440 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107225.log 2026-03-31T21:28:00.441 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107196.log: 29.6% -- replaced with /var/log/ceph/ceph-client.admin.107196.log.gz 2026-03-31T21:28:00.441 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107261.log 2026-03-31T21:28:00.441 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107225.log.gz 2026-03-31T21:28:00.442 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107309.log 2026-03-31T21:28:00.442 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107261.log.gz 2026-03-31T21:28:00.443 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107392.log 2026-03-31T21:28:00.443 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107309.log.gz 2026-03-31T21:28:00.443 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107521.log 2026-03-31T21:28:00.443 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107392.log.gz 2026-03-31T21:28:00.444 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107520.log 2026-03-31T21:28:00.444 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107521.log: 30.3% -- replaced with /var/log/ceph/ceph-client.admin.107521.log.gz 2026-03-31T21:28:00.444 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107519.log 2026-03-31T21:28:00.445 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107520.log.gz 2026-03-31T21:28:00.445 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107522.log 2026-03-31T21:28:00.445 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107519.log.gz 2026-03-31T21:28:00.446 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107679.log 2026-03-31T21:28:00.446 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107522.log.gz 2026-03-31T21:28:00.446 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107678.log 2026-03-31T21:28:00.447 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107679.log.gz 2026-03-31T21:28:00.447 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107677.log 2026-03-31T21:28:00.448 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107678.log.gz 2026-03-31T21:28:00.448 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107784.log 2026-03-31T21:28:00.448 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107677.log.gz 2026-03-31T21:28:00.449 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107860.log 2026-03-31T21:28:00.449 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107784.log.gz 2026-03-31T21:28:00.449 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107859.log 2026-03-31T21:28:00.450 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107860.log.gz 2026-03-31T21:28:00.450 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107890.log 2026-03-31T21:28:00.450 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107859.log.gz 2026-03-31T21:28:00.451 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107978.log 2026-03-31T21:28:00.451 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107890.log.gz 2026-03-31T21:28:00.451 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108029.log 2026-03-31T21:28:00.452 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.107978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107978.log.gz 2026-03-31T21:28:00.452 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108078.log 2026-03-31T21:28:00.452 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108029.log.gz 2026-03-31T21:28:00.453 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108207.log 2026-03-31T21:28:00.453 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108078.log.gz 2026-03-31T21:28:00.453 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108208.log 2026-03-31T21:28:00.454 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108207.log.gz 2026-03-31T21:28:00.454 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108205.log 2026-03-31T21:28:00.454 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108208.log.gz 2026-03-31T21:28:00.455 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108206.log 2026-03-31T21:28:00.455 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108205.log.gz 2026-03-31T21:28:00.455 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108374.log 2026-03-31T21:28:00.456 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108206.log.gz 2026-03-31T21:28:00.456 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108373.log 2026-03-31T21:28:00.456 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108374.log.gz 2026-03-31T21:28:00.456 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108377.log 2026-03-31T21:28:00.457 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108373.log.gz 2026-03-31T21:28:00.457 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108464.log 2026-03-31T21:28:00.458 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108377.log.gz 2026-03-31T21:28:00.458 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108519.log 2026-03-31T21:28:00.458 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108464.log.gz 2026-03-31T21:28:00.458 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108582.log 2026-03-31T21:28:00.459 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108519.log.gz 2026-03-31T21:28:00.459 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108583.log 2026-03-31T21:28:00.460 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108582.log: 27.2% -- replaced with /var/log/ceph/ceph-client.admin.108582.log.gz 2026-03-31T21:28:00.460 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108623.log 2026-03-31T21:28:00.460 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108583.log.gz 2026-03-31T21:28:00.460 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108715.log 2026-03-31T21:28:00.461 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108623.log.gz 2026-03-31T21:28:00.461 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108842.log 2026-03-31T21:28:00.461 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108715.log.gz 2026-03-31T21:28:00.462 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108845.log 2026-03-31T21:28:00.462 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108842.log.gz 2026-03-31T21:28:00.462 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108843.log 2026-03-31T21:28:00.463 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108845.log.gz 2026-03-31T21:28:00.463 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108844.log 2026-03-31T21:28:00.463 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108843.log.gz 2026-03-31T21:28:00.464 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109037.log 2026-03-31T21:28:00.464 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.108844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108844.log.gz 2026-03-31T21:28:00.464 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109040.log 2026-03-31T21:28:00.465 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109037.log.gz 2026-03-31T21:28:00.465 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109035.log 2026-03-31T21:28:00.465 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109040.log.gz 2026-03-31T21:28:00.466 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109041.log 2026-03-31T21:28:00.466 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109035.log.gz 2026-03-31T21:28:00.466 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109183.log 2026-03-31T21:28:00.467 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109041.log.gz 2026-03-31T21:28:00.467 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109182.log 2026-03-31T21:28:00.467 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109183.log.gz 2026-03-31T21:28:00.468 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109257.log 2026-03-31T21:28:00.468 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109182.log.gz 2026-03-31T21:28:00.468 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109256.log 2026-03-31T21:28:00.469 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109257.log.gz 2026-03-31T21:28:00.469 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109352.log 2026-03-31T21:28:00.469 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109256.log.gz 2026-03-31T21:28:00.470 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109481.log 2026-03-31T21:28:00.470 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109352.log.gz 2026-03-31T21:28:00.471 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109479.log 2026-03-31T21:28:00.471 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109481.log.gz 2026-03-31T21:28:00.471 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109482.log 2026-03-31T21:28:00.471 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109479.log.gz 2026-03-31T21:28:00.472 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109480.log 2026-03-31T21:28:00.472 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109482.log.gz 2026-03-31T21:28:00.472 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109628.log 2026-03-31T21:28:00.473 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109480.log.gz 2026-03-31T21:28:00.473 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109677.log 2026-03-31T21:28:00.473 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109628.log.gz 2026-03-31T21:28:00.474 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109672.log 2026-03-31T21:28:00.474 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109677.log.gz 2026-03-31T21:28:00.474 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109678.log 2026-03-31T21:28:00.475 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109672.log.gz 2026-03-31T21:28:00.475 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109793.log 2026-03-31T21:28:00.475 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109678.log.gz 2026-03-31T21:28:00.476 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109872.log 2026-03-31T21:28:00.476 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109793.log.gz 2026-03-31T21:28:00.476 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109869.log 2026-03-31T21:28:00.477 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109872.log.gz 2026-03-31T21:28:00.477 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109884.log 2026-03-31T21:28:00.477 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109869.log.gz 2026-03-31T21:28:00.477 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109989.log 2026-03-31T21:28:00.478 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109884.log.gz 2026-03-31T21:28:00.478 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110115.log 2026-03-31T21:28:00.478 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.109989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109989.log.gz 2026-03-31T21:28:00.479 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110116.log 2026-03-31T21:28:00.479 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110115.log.gz 2026-03-31T21:28:00.479 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110114.log 2026-03-31T21:28:00.480 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110116.log.gz 2026-03-31T21:28:00.480 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110119.log 2026-03-31T21:28:00.480 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110114.log.gz 2026-03-31T21:28:00.481 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110303.log 2026-03-31T21:28:00.481 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110119.log.gz 2026-03-31T21:28:00.481 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110314.log 2026-03-31T21:28:00.482 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110303.log.gz 2026-03-31T21:28:00.482 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110313.log 2026-03-31T21:28:00.482 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110314.log.gz 2026-03-31T21:28:00.483 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110315.log 2026-03-31T21:28:00.483 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110313.log.gz 2026-03-31T21:28:00.483 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110442.log 2026-03-31T21:28:00.484 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110315.log.gz 2026-03-31T21:28:00.484 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110521.log 2026-03-31T21:28:00.484 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110442.log.gz 2026-03-31T21:28:00.485 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110522.log 2026-03-31T21:28:00.485 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110521.log.gz 2026-03-31T21:28:00.486 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110523.log 2026-03-31T21:28:00.487 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110522.log.gz 2026-03-31T21:28:00.487 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110638.log 2026-03-31T21:28:00.487 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110523.log.gz 2026-03-31T21:28:00.488 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110765.log 2026-03-31T21:28:00.488 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110638.log.gz 2026-03-31T21:28:00.488 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110768.log 2026-03-31T21:28:00.489 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110765.log.gz 2026-03-31T21:28:00.489 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110766.log 2026-03-31T21:28:00.489 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110768.log.gz 2026-03-31T21:28:00.490 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110767.log 2026-03-31T21:28:00.490 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110766.log.gz 2026-03-31T21:28:00.490 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110936.log 2026-03-31T21:28:00.491 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110767.log.gz 2026-03-31T21:28:00.491 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110929.log 2026-03-31T21:28:00.491 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110936.log.gz 2026-03-31T21:28:00.492 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110937.log 2026-03-31T21:28:00.492 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110929.log.gz 2026-03-31T21:28:00.492 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111024.log 2026-03-31T21:28:00.493 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.110937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110937.log.gz 2026-03-31T21:28:00.493 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111132.log 2026-03-31T21:28:00.493 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111024.log.gz 2026-03-31T21:28:00.494 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111133.log 2026-03-31T21:28:00.494 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111132.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111132.log.gz 2026-03-31T21:28:00.494 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111131.log 2026-03-31T21:28:00.495 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111133.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111133.log.gz 2026-03-31T21:28:00.495 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111220.log 2026-03-31T21:28:00.495 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111131.log.gz 2026-03-31T21:28:00.496 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111275.log 2026-03-31T21:28:00.496 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111220.log.gz 2026-03-31T21:28:00.496 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111398.log 2026-03-31T21:28:00.497 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111275.log.gz 2026-03-31T21:28:00.497 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111397.log 2026-03-31T21:28:00.497 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111398.log.gz 2026-03-31T21:28:00.498 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111404.log 2026-03-31T21:28:00.498 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111397.log.gz 2026-03-31T21:28:00.498 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111405.log 2026-03-31T21:28:00.499 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111404.log.gz 2026-03-31T21:28:00.499 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111563.log 2026-03-31T21:28:00.499 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111405.log.gz 2026-03-31T21:28:00.500 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111598.log 2026-03-31T21:28:00.500 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111563.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111563.log.gz 2026-03-31T21:28:00.500 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111597.log 2026-03-31T21:28:00.501 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111598.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111598.log.gz 2026-03-31T21:28:00.501 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111601.log 2026-03-31T21:28:00.501 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111597.log.gz 2026-03-31T21:28:00.502 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111742.log 2026-03-31T21:28:00.502 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111601.log.gz 2026-03-31T21:28:00.502 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111743.log 2026-03-31T21:28:00.503 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111742.log.gz 2026-03-31T21:28:00.503 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111821.log 2026-03-31T21:28:00.503 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111743.log.gz 2026-03-31T21:28:00.504 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111827.log 2026-03-31T21:28:00.504 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111821.log.gz 2026-03-31T21:28:00.504 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111912.log 2026-03-31T21:28:00.505 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111827.log.gz 2026-03-31T21:28:00.505 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111976.log 2026-03-31T21:28:00.505 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111912.log.gz 2026-03-31T21:28:00.506 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111975.log 2026-03-31T21:28:00.506 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111976.log.gz 2026-03-31T21:28:00.506 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112015.log 2026-03-31T21:28:00.507 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.111975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111975.log.gz 2026-03-31T21:28:00.507 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112108.log 2026-03-31T21:28:00.507 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112015.log.gz 2026-03-31T21:28:00.508 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112234.log 2026-03-31T21:28:00.508 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112108.log.gz 2026-03-31T21:28:00.508 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112235.log 2026-03-31T21:28:00.509 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112234.log.gz 2026-03-31T21:28:00.509 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112233.log 2026-03-31T21:28:00.509 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112235.log.gz 2026-03-31T21:28:00.510 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112238.log 2026-03-31T21:28:00.510 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112233.log.gz 2026-03-31T21:28:00.510 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112353.log 2026-03-31T21:28:00.511 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112238.log.gz 2026-03-31T21:28:00.511 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112424.log 2026-03-31T21:28:00.511 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112353.log.gz 2026-03-31T21:28:00.512 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112427.log 2026-03-31T21:28:00.512 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112424.log.gz 2026-03-31T21:28:00.512 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112455.log 2026-03-31T21:28:00.513 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112427.log.gz 2026-03-31T21:28:00.513 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112549.log 2026-03-31T21:28:00.513 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112455.log.gz 2026-03-31T21:28:00.514 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112596.log 2026-03-31T21:28:00.514 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112549.log.gz 2026-03-31T21:28:00.514 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112648.log 2026-03-31T21:28:00.515 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112596.log.gz 2026-03-31T21:28:00.515 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112660.log 2026-03-31T21:28:00.515 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112648.log.gz 2026-03-31T21:28:00.516 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112745.log 2026-03-31T21:28:00.516 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112660.log.gz 2026-03-31T21:28:00.516 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112792.log 2026-03-31T21:28:00.517 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112745.log.gz 2026-03-31T21:28:00.517 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112845.log 2026-03-31T21:28:00.517 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112792.log.gz 2026-03-31T21:28:00.518 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112856.log 2026-03-31T21:28:00.518 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112845.log.gz 2026-03-31T21:28:00.518 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112941.log 2026-03-31T21:28:00.519 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112856.log.gz 2026-03-31T21:28:00.519 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113069.log 2026-03-31T21:28:00.519 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.112941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112941.log.gz 2026-03-31T21:28:00.520 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113070.log 2026-03-31T21:28:00.520 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113069.log.gz 2026-03-31T21:28:00.520 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113071.log 2026-03-31T21:28:00.521 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113070.log: 29.6% -- replaced with /var/log/ceph/ceph-client.admin.113070.log.gz 2026-03-31T21:28:00.521 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113067.log 2026-03-31T21:28:00.521 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113071.log.gz 2026-03-31T21:28:00.522 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113236.log 2026-03-31T21:28:00.522 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113067.log.gz 2026-03-31T21:28:00.522 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113235.log 2026-03-31T21:28:00.523 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113236.log.gz 2026-03-31T21:28:00.523 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113234.log 2026-03-31T21:28:00.523 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113235.log.gz 2026-03-31T21:28:00.524 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113333.log 2026-03-31T21:28:00.524 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113234.log.gz 2026-03-31T21:28:00.524 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113382.log 2026-03-31T21:28:00.525 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113333.log.gz 2026-03-31T21:28:00.525 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113437.log 2026-03-31T21:28:00.525 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113382.log.gz 2026-03-31T21:28:00.525 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113438.log 2026-03-31T21:28:00.526 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113437.log.gz 2026-03-31T21:28:00.526 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113525.log 2026-03-31T21:28:00.526 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113438.log.gz 2026-03-31T21:28:00.527 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113578.log 2026-03-31T21:28:00.527 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113525.log.gz 2026-03-31T21:28:00.527 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113706.log 2026-03-31T21:28:00.528 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113578.log.gz 2026-03-31T21:28:00.528 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113705.log 2026-03-31T21:28:00.528 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113706.log.gz 2026-03-31T21:28:00.529 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113708.log 2026-03-31T21:28:00.529 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113705.log.gz 2026-03-31T21:28:00.529 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113707.log 2026-03-31T21:28:00.530 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113708.log.gz 2026-03-31T21:28:00.530 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113845.log 2026-03-31T21:28:00.530 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113707.log.gz 2026-03-31T21:28:00.531 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113846.log 2026-03-31T21:28:00.531 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113845.log.gz 2026-03-31T21:28:00.532 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113948.log 2026-03-31T21:28:00.532 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113846.log.gz 2026-03-31T21:28:00.532 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113946.log 2026-03-31T21:28:00.533 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113948.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113948.log.gz 2026-03-31T21:28:00.533 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114046.log 2026-03-31T21:28:00.533 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.113946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113946.log.gz 2026-03-31T21:28:00.533 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114045.log 2026-03-31T21:28:00.534 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114046.log.gz 2026-03-31T21:28:00.534 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114141.log 2026-03-31T21:28:00.534 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114045.log.gz 2026-03-31T21:28:00.535 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114144.log 2026-03-31T21:28:00.535 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114141.log.gz 2026-03-31T21:28:00.535 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114215.log 2026-03-31T21:28:00.536 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114144.log.gz 2026-03-31T21:28:00.536 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114342.log 2026-03-31T21:28:00.536 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114215.log.gz 2026-03-31T21:28:00.537 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114343.log 2026-03-31T21:28:00.537 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114342.log.gz 2026-03-31T21:28:00.537 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114340.log 2026-03-31T21:28:00.538 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114343.log.gz 2026-03-31T21:28:00.538 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114345.log 2026-03-31T21:28:00.538 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114340.log.gz 2026-03-31T21:28:00.539 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114524.log 2026-03-31T21:28:00.539 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114345.log.gz 2026-03-31T21:28:00.539 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114539.log 2026-03-31T21:28:00.540 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114524.log.gz 2026-03-31T21:28:00.540 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114537.log 2026-03-31T21:28:00.540 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114539.log.gz 2026-03-31T21:28:00.541 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114541.log 2026-03-31T21:28:00.541 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114537.log.gz 2026-03-31T21:28:00.541 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114708.log 2026-03-31T21:28:00.542 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114541.log.gz 2026-03-31T21:28:00.542 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114709.log 2026-03-31T21:28:00.542 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114708.log.gz 2026-03-31T21:28:00.543 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114710.log 2026-03-31T21:28:00.543 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114709.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114709.log.gz 2026-03-31T21:28:00.544 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114797.log 2026-03-31T21:28:00.545 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114710.log.gz 2026-03-31T21:28:00.545 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114852.log 2026-03-31T21:28:00.545 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114797.log.gz 2026-03-31T21:28:00.546 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114906.log 2026-03-31T21:28:00.546 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114852.log.gz 2026-03-31T21:28:00.546 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114908.log 2026-03-31T21:28:00.547 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114906.log.gz 2026-03-31T21:28:00.547 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114983.log 2026-03-31T21:28:00.547 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114908.log.gz 2026-03-31T21:28:00.548 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115048.log 2026-03-31T21:28:00.548 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.114983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114983.log.gz 2026-03-31T21:28:00.548 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115178.log 2026-03-31T21:28:00.549 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115048.log.gz 2026-03-31T21:28:00.549 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115177.log 2026-03-31T21:28:00.549 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115178.log.gz 2026-03-31T21:28:00.550 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115175.log 2026-03-31T21:28:00.550 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115177.log.gz 2026-03-31T21:28:00.550 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115176.log 2026-03-31T21:28:00.551 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115175.log.gz 2026-03-31T21:28:00.551 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115363.log 2026-03-31T21:28:00.551 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115176.log.gz 2026-03-31T21:28:00.552 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115373.log 2026-03-31T21:28:00.552 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115363.log.gz 2026-03-31T21:28:00.552 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115371.log 2026-03-31T21:28:00.553 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115373.log.gz 2026-03-31T21:28:00.553 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115374.log 2026-03-31T21:28:00.553 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115371.log.gz 2026-03-31T21:28:00.554 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115489.log 2026-03-31T21:28:00.554 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115374.log.gz 2026-03-31T21:28:00.554 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115587.log 2026-03-31T21:28:00.555 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115489.log.gz 2026-03-31T21:28:00.555 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115583.log 2026-03-31T21:28:00.555 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115587.log.gz 2026-03-31T21:28:00.556 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115588.log 2026-03-31T21:28:00.556 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115583.log.gz 2026-03-31T21:28:00.556 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115685.log 2026-03-31T21:28:00.557 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115588.log.gz 2026-03-31T21:28:00.557 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115712.log 2026-03-31T21:28:00.557 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115685.log.gz 2026-03-31T21:28:00.557 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115783.log 2026-03-31T21:28:00.558 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115712.log.gz 2026-03-31T21:28:00.558 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115913.log 2026-03-31T21:28:00.558 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115783.log.gz 2026-03-31T21:28:00.559 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115909.log 2026-03-31T21:28:00.559 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115913.log.gz 2026-03-31T21:28:00.559 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115914.log 2026-03-31T21:28:00.560 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115909.log.gz 2026-03-31T21:28:00.560 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115912.log 2026-03-31T21:28:00.560 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115914.log.gz 2026-03-31T21:28:00.561 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116070.log 2026-03-31T21:28:00.561 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.115912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115912.log.gz 2026-03-31T21:28:00.561 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116082.log 2026-03-31T21:28:00.562 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116070.log.gz 2026-03-31T21:28:00.562 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116083.log 2026-03-31T21:28:00.562 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116082.log.gz 2026-03-31T21:28:00.563 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116150.log 2026-03-31T21:28:00.563 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116083.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116083.log.gz 2026-03-31T21:28:00.563 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116225.log 2026-03-31T21:28:00.564 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116150.log.gz 2026-03-31T21:28:00.564 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116281.log 2026-03-31T21:28:00.564 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116225.log.gz 2026-03-31T21:28:00.565 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116280.log 2026-03-31T21:28:00.565 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116281.log.gz 2026-03-31T21:28:00.565 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116347.log 2026-03-31T21:28:00.566 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116280.log.gz 2026-03-31T21:28:00.566 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116421.log 2026-03-31T21:28:00.566 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116347.log.gz 2026-03-31T21:28:00.567 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116470.log 2026-03-31T21:28:00.567 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116421.log.gz 2026-03-31T21:28:00.567 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116596.log 2026-03-31T21:28:00.568 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116470.log.gz 2026-03-31T21:28:00.568 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116594.log 2026-03-31T21:28:00.568 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116596.log.gz 2026-03-31T21:28:00.569 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116599.log 2026-03-31T21:28:00.569 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116594.log.gz 2026-03-31T21:28:00.569 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116600.log 2026-03-31T21:28:00.570 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116599.log.gz 2026-03-31T21:28:00.570 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116769.log 2026-03-31T21:28:00.570 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116600.log.gz 2026-03-31T21:28:00.571 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116763.log 2026-03-31T21:28:00.571 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116769.log.gz 2026-03-31T21:28:00.571 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116768.log 2026-03-31T21:28:00.572 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116763.log.gz 2026-03-31T21:28:00.572 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116856.log 2026-03-31T21:28:00.572 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116768.log.gz 2026-03-31T21:28:00.573 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116938.log 2026-03-31T21:28:00.573 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116856.log.gz 2026-03-31T21:28:00.573 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116937.log 2026-03-31T21:28:00.574 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116938.log.gz 2026-03-31T21:28:00.574 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117002.log 2026-03-31T21:28:00.574 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.116937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116937.log.gz 2026-03-31T21:28:00.575 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117033.log 2026-03-31T21:28:00.575 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117002.log.gz 2026-03-31T21:28:00.575 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117107.log 2026-03-31T21:28:00.576 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117033.log.gz 2026-03-31T21:28:00.576 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117135.log 2026-03-31T21:28:00.576 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117107.log.gz 2026-03-31T21:28:00.577 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117188.log 2026-03-31T21:28:00.577 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117135.log.gz 2026-03-31T21:28:00.577 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117229.log 2026-03-31T21:28:00.578 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117188.log.gz 2026-03-31T21:28:00.578 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117303.log 2026-03-31T21:28:00.578 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117229.log.gz 2026-03-31T21:28:00.578 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117432.log 2026-03-31T21:28:00.579 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117303.log.gz 2026-03-31T21:28:00.579 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117431.log 2026-03-31T21:28:00.579 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117432.log.gz 2026-03-31T21:28:00.580 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117434.log 2026-03-31T21:28:00.580 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117431.log.gz 2026-03-31T21:28:00.580 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117436.log 2026-03-31T21:28:00.581 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117434.log.gz 2026-03-31T21:28:00.581 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117579.log 2026-03-31T21:28:00.581 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117436.log.gz 2026-03-31T21:28:00.582 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117629.log 2026-03-31T21:28:00.582 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117579.log.gz 2026-03-31T21:28:00.582 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117628.log 2026-03-31T21:28:00.583 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117629.log.gz 2026-03-31T21:28:00.583 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117632.log 2026-03-31T21:28:00.583 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117628.log.gz 2026-03-31T21:28:00.584 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117747.log 2026-03-31T21:28:00.584 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117632.log.gz 2026-03-31T21:28:00.584 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117829.log 2026-03-31T21:28:00.585 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117747.log.gz 2026-03-31T21:28:00.585 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117830.log 2026-03-31T21:28:00.585 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117829.log.gz 2026-03-31T21:28:00.586 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117828.log 2026-03-31T21:28:00.586 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117830.log.gz 2026-03-31T21:28:00.586 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117943.log 2026-03-31T21:28:00.587 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117828.log.gz 2026-03-31T21:28:00.587 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117992.log 2026-03-31T21:28:00.587 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117943.log.gz 2026-03-31T21:28:00.587 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118122.log 2026-03-31T21:28:00.588 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.117992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117992.log.gz 2026-03-31T21:28:00.588 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118118.log 2026-03-31T21:28:00.589 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118122.log.gz 2026-03-31T21:28:00.589 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118121.log 2026-03-31T21:28:00.589 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118118.log.gz 2026-03-31T21:28:00.590 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118119.log 2026-03-31T21:28:00.590 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118121.log.gz 2026-03-31T21:28:00.590 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118291.log 2026-03-31T21:28:00.591 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118119.log.gz 2026-03-31T21:28:00.591 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118290.log 2026-03-31T21:28:00.591 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118291.log.gz 2026-03-31T21:28:00.592 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118289.log 2026-03-31T21:28:00.592 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118290.log.gz 2026-03-31T21:28:00.592 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118378.log 2026-03-31T21:28:00.593 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118289.log.gz 2026-03-31T21:28:00.593 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118486.log 2026-03-31T21:28:00.593 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118378.log.gz 2026-03-31T21:28:00.594 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118487.log 2026-03-31T21:28:00.594 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118486.log.gz 2026-03-31T21:28:00.594 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118485.log 2026-03-31T21:28:00.595 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118487.log.gz 2026-03-31T21:28:00.595 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118574.log 2026-03-31T21:28:00.595 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118485.log.gz 2026-03-31T21:28:00.596 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118655.log 2026-03-31T21:28:00.596 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118574.log.gz 2026-03-31T21:28:00.596 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118656.log 2026-03-31T21:28:00.596 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118655.log.gz 2026-03-31T21:28:00.597 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118720.log 2026-03-31T21:28:00.597 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118656.log.gz 2026-03-31T21:28:00.597 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118776.log 2026-03-31T21:28:00.598 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118720.log.gz 2026-03-31T21:28:00.598 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118903.log 2026-03-31T21:28:00.598 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118776.log.gz 2026-03-31T21:28:00.599 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118904.log 2026-03-31T21:28:00.599 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118903.log.gz 2026-03-31T21:28:00.599 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118906.log 2026-03-31T21:28:00.600 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118904.log.gz 2026-03-31T21:28:00.600 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118902.log 2026-03-31T21:28:00.600 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118906.log.gz 2026-03-31T21:28:00.601 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119074.log 2026-03-31T21:28:00.601 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.118902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118902.log.gz 2026-03-31T21:28:00.601 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119075.log 2026-03-31T21:28:00.602 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119074.log.gz 2026-03-31T21:28:00.602 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119073.log 2026-03-31T21:28:00.602 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119075.log.gz 2026-03-31T21:28:00.603 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119107.log 2026-03-31T21:28:00.603 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119073.log.gz 2026-03-31T21:28:00.603 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119217.log 2026-03-31T21:28:00.604 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119107.log.gz 2026-03-31T21:28:00.604 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119271.log 2026-03-31T21:28:00.604 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119217.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.119217.log.gz 2026-03-31T21:28:00.605 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119278.log 2026-03-31T21:28:00.605 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119271.log.gz 2026-03-31T21:28:00.605 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119314.log 2026-03-31T21:28:00.606 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119278.log.gz 2026-03-31T21:28:00.606 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119413.log 2026-03-31T21:28:00.606 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119314.log.gz 2026-03-31T21:28:00.607 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119539.log 2026-03-31T21:28:00.607 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119413.log.gz 2026-03-31T21:28:00.607 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119540.log 2026-03-31T21:28:00.608 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119539.log.gz 2026-03-31T21:28:00.608 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119538.log 2026-03-31T21:28:00.608 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119540.log.gz 2026-03-31T21:28:00.609 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119543.log 2026-03-31T21:28:00.609 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119538.log.gz 2026-03-31T21:28:00.609 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119712.log 2026-03-31T21:28:00.610 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119543.log.gz 2026-03-31T21:28:00.610 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119739.log 2026-03-31T21:28:00.610 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119712.log.gz 2026-03-31T21:28:00.611 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119734.log 2026-03-31T21:28:00.611 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119739.log.gz 2026-03-31T21:28:00.615 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119738.log 2026-03-31T21:28:00.616 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119734.log.gz 2026-03-31T21:28:00.616 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119854.log 2026-03-31T21:28:00.616 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119738.log.gz 2026-03-31T21:28:00.617 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119933.log 2026-03-31T21:28:00.617 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119854.log.gz 2026-03-31T21:28:00.617 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119934.log 2026-03-31T21:28:00.618 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119933.log.gz 2026-03-31T21:28:00.618 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119942.log 2026-03-31T21:28:00.618 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119934.log.gz 2026-03-31T21:28:00.619 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120050.log 2026-03-31T21:28:00.619 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.119942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119942.log.gz 2026-03-31T21:28:00.619 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120178.log 2026-03-31T21:28:00.620 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120050.log.gz 2026-03-31T21:28:00.620 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120180.log 2026-03-31T21:28:00.620 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120178.log.gz 2026-03-31T21:28:00.621 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120179.log 2026-03-31T21:28:00.621 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120180.log.gz 2026-03-31T21:28:00.621 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120177.log 2026-03-31T21:28:00.622 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120179.log.gz 2026-03-31T21:28:00.622 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120367.log 2026-03-31T21:28:00.622 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120177.log.gz 2026-03-31T21:28:00.623 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120368.log 2026-03-31T21:28:00.623 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120367.log.gz 2026-03-31T21:28:00.623 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120376.log 2026-03-31T21:28:00.624 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120368.log.gz 2026-03-31T21:28:00.624 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120375.log 2026-03-31T21:28:00.624 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120376.log.gz 2026-03-31T21:28:00.624 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120517.log 2026-03-31T21:28:00.625 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120375.log.gz 2026-03-31T21:28:00.625 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120518.log 2026-03-31T21:28:00.625 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120517.log.gz 2026-03-31T21:28:00.626 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120573.log 2026-03-31T21:28:00.626 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120518.log.gz 2026-03-31T21:28:00.626 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120592.log 2026-03-31T21:28:00.627 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120573.log.gz 2026-03-31T21:28:00.627 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120687.log 2026-03-31T21:28:00.627 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120592.log.gz 2026-03-31T21:28:00.628 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120716.log 2026-03-31T21:28:00.628 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120687.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120687.log.gz 2026-03-31T21:28:00.628 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120786.log 2026-03-31T21:28:00.629 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120716.log.gz 2026-03-31T21:28:00.629 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120795.log 2026-03-31T21:28:00.629 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120786.log.gz 2026-03-31T21:28:00.630 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120883.log 2026-03-31T21:28:00.630 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120795.log.gz 2026-03-31T21:28:00.630 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121010.log 2026-03-31T21:28:00.631 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.120883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120883.log.gz 2026-03-31T21:28:00.631 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121012.log 2026-03-31T21:28:00.631 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121010.log.gz 2026-03-31T21:28:00.632 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121013.log 2026-03-31T21:28:00.632 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121012.log.gz 2026-03-31T21:28:00.632 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121009.log 2026-03-31T21:28:00.633 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121013.log.gz 2026-03-31T21:28:00.633 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121179.log 2026-03-31T21:28:00.633 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121009.log.gz 2026-03-31T21:28:00.634 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121177.log 2026-03-31T21:28:00.634 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121179.log.gz 2026-03-31T21:28:00.634 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121182.log 2026-03-31T21:28:00.635 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121177.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121177.log.gz 2026-03-31T21:28:00.635 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121269.log 2026-03-31T21:28:00.635 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121182.log.gz 2026-03-31T21:28:00.636 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121350.log 2026-03-31T21:28:00.636 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121269.log.gz 2026-03-31T21:28:00.636 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121351.log 2026-03-31T21:28:00.637 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121350.log.gz 2026-03-31T21:28:00.637 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121379.log 2026-03-31T21:28:00.637 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121351.log.gz 2026-03-31T21:28:00.638 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121446.log 2026-03-31T21:28:00.638 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121379.log.gz 2026-03-31T21:28:00.638 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121520.log 2026-03-31T21:28:00.639 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121446.log.gz 2026-03-31T21:28:00.639 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121650.log 2026-03-31T21:28:00.639 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121520.log.gz 2026-03-31T21:28:00.640 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121645.log 2026-03-31T21:28:00.640 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121650.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121650.log.gz 2026-03-31T21:28:00.640 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121649.log 2026-03-31T21:28:00.641 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121645.log.gz 2026-03-31T21:28:00.641 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121648.log 2026-03-31T21:28:00.641 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121649.log.gz 2026-03-31T21:28:00.642 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121842.log 2026-03-31T21:28:00.642 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121648.log.gz 2026-03-31T21:28:00.642 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121845.log 2026-03-31T21:28:00.643 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121842.log.gz 2026-03-31T21:28:00.643 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121844.log 2026-03-31T21:28:00.643 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121845.log.gz 2026-03-31T21:28:00.644 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121846.log 2026-03-31T21:28:00.644 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121844.log.gz 2026-03-31T21:28:00.644 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121961.log 2026-03-31T21:28:00.645 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121846.log.gz 2026-03-31T21:28:00.645 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122044.log 2026-03-31T21:28:00.645 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.121961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121961.log.gz 2026-03-31T21:28:00.646 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122042.log 2026-03-31T21:28:00.646 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122044.log.gz 2026-03-31T21:28:00.646 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122041.log 2026-03-31T21:28:00.647 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122042.log.gz 2026-03-31T21:28:00.647 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122157.log 2026-03-31T21:28:00.647 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122041.log.gz 2026-03-31T21:28:00.647 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122286.log 2026-03-31T21:28:00.648 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122157.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122157.log.gz 2026-03-31T21:28:00.648 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122285.log 2026-03-31T21:28:00.648 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122286.log.gz 2026-03-31T21:28:00.649 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122287.log 2026-03-31T21:28:00.649 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122285.log.gz 2026-03-31T21:28:00.649 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122284.log 2026-03-31T21:28:00.650 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122287.log.gz 2026-03-31T21:28:00.650 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122455.log 2026-03-31T21:28:00.650 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122284.log.gz 2026-03-31T21:28:00.651 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122454.log 2026-03-31T21:28:00.651 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122455.log.gz 2026-03-31T21:28:00.651 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122456.log 2026-03-31T21:28:00.652 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122454.log.gz 2026-03-31T21:28:00.652 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122541.log 2026-03-31T21:28:00.652 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122456.log.gz 2026-03-31T21:28:00.653 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122652.log 2026-03-31T21:28:00.653 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122541.log.gz 2026-03-31T21:28:00.653 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122649.log 2026-03-31T21:28:00.654 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122652.log.gz 2026-03-31T21:28:00.654 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122651.log 2026-03-31T21:28:00.654 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122649.log.gz 2026-03-31T21:28:00.655 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122738.log 2026-03-31T21:28:00.655 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122651.log.gz 2026-03-31T21:28:00.655 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122794.log 2026-03-31T21:28:00.656 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122738.log.gz 2026-03-31T21:28:00.656 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122850.log 2026-03-31T21:28:00.656 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122794.log.gz 2026-03-31T21:28:00.657 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122849.log 2026-03-31T21:28:00.657 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122850.log.gz 2026-03-31T21:28:00.657 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122897.log 2026-03-31T21:28:00.658 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122849.log: 27.6% -- replaced with /var/log/ceph/ceph-client.admin.122849.log.gz 2026-03-31T21:28:00.658 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122990.log 2026-03-31T21:28:00.658 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122897.log.gz 2026-03-31T21:28:00.659 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123119.log 2026-03-31T21:28:00.659 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.122990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.122990.log.gz 2026-03-31T21:28:00.659 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123120.log 2026-03-31T21:28:00.660 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123119.log.gz 2026-03-31T21:28:00.660 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123117.log 2026-03-31T21:28:00.660 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123120.log.gz 2026-03-31T21:28:00.661 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123118.log 2026-03-31T21:28:00.661 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123117.log.gz 2026-03-31T21:28:00.661 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123289.log 2026-03-31T21:28:00.662 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123118.log.gz 2026-03-31T21:28:00.662 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123288.log 2026-03-31T21:28:00.662 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123289.log.gz 2026-03-31T21:28:00.663 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123287.log 2026-03-31T21:28:00.663 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123288.log.gz 2026-03-31T21:28:00.663 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123338.log 2026-03-31T21:28:00.664 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123287.log.gz 2026-03-31T21:28:00.664 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123431.log 2026-03-31T21:28:00.664 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123338.log.gz 2026-03-31T21:28:00.665 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123486.log 2026-03-31T21:28:00.665 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123431.log.gz 2026-03-31T21:28:00.665 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123485.log 2026-03-31T21:28:00.666 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123486.log.gz 2026-03-31T21:28:00.666 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123543.log 2026-03-31T21:28:00.666 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123485.log.gz 2026-03-31T21:28:00.667 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123627.log 2026-03-31T21:28:00.667 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123543.log.gz 2026-03-31T21:28:00.667 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123757.log 2026-03-31T21:28:00.668 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123627.log.gz 2026-03-31T21:28:00.668 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123756.log 2026-03-31T21:28:00.668 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123757.log.gz 2026-03-31T21:28:00.669 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123753.log 2026-03-31T21:28:00.669 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123756.log.gz 2026-03-31T21:28:00.669 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123755.log 2026-03-31T21:28:00.670 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123753.log.gz 2026-03-31T21:28:00.670 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123916.log 2026-03-31T21:28:00.670 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123755.log.gz 2026-03-31T21:28:00.670 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123922.log 2026-03-31T21:28:00.671 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123916.log.gz 2026-03-31T21:28:00.671 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123926.log 2026-03-31T21:28:00.671 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123922.log.gz 2026-03-31T21:28:00.672 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124013.log 2026-03-31T21:28:00.672 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.123926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.123926.log.gz 2026-03-31T21:28:00.672 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124068.log 2026-03-31T21:28:00.673 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124013.log.gz 2026-03-31T21:28:00.673 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124122.log 2026-03-31T21:28:00.673 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124068.log.gz 2026-03-31T21:28:00.674 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124124.log 2026-03-31T21:28:00.674 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124122.log.gz 2026-03-31T21:28:00.674 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124208.log 2026-03-31T21:28:00.675 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124124.log.gz 2026-03-31T21:28:00.675 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124264.log 2026-03-31T21:28:00.675 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124208.log.gz 2026-03-31T21:28:00.676 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124395.log 2026-03-31T21:28:00.676 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124264.log.gz 2026-03-31T21:28:00.676 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124393.log 2026-03-31T21:28:00.677 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124395.log.gz 2026-03-31T21:28:00.677 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124391.log 2026-03-31T21:28:00.677 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124393.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124393.log.gz 2026-03-31T21:28:00.678 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124394.log 2026-03-31T21:28:00.678 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124391.log.gz 2026-03-31T21:28:00.678 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124508.log 2026-03-31T21:28:00.679 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124394.log.gz 2026-03-31T21:28:00.679 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124568.log 2026-03-31T21:28:00.679 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124508.log.gz 2026-03-31T21:28:00.680 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124569.log 2026-03-31T21:28:00.680 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124568.log.gz 2026-03-31T21:28:00.680 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124653.log 2026-03-31T21:28:00.681 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124569.log.gz 2026-03-31T21:28:00.681 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124706.log 2026-03-31T21:28:00.681 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124653.log.gz 2026-03-31T21:28:00.682 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124769.log 2026-03-31T21:28:00.682 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124706.log.gz 2026-03-31T21:28:00.682 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124770.log 2026-03-31T21:28:00.683 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124769.log.gz 2026-03-31T21:28:00.683 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124849.log 2026-03-31T21:28:00.683 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124770.log.gz 2026-03-31T21:28:00.684 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124902.log 2026-03-31T21:28:00.684 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124849.log.gz 2026-03-31T21:28:00.684 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124966.log 2026-03-31T21:28:00.685 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124902.log.gz 2026-03-31T21:28:00.685 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124964.log 2026-03-31T21:28:00.685 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124966.log.gz 2026-03-31T21:28:00.686 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125045.log 2026-03-31T21:28:00.686 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.124964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.124964.log.gz 2026-03-31T21:28:00.686 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125098.log 2026-03-31T21:28:00.687 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125045.log.gz 2026-03-31T21:28:00.687 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125225.log 2026-03-31T21:28:00.687 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125098.log.gz 2026-03-31T21:28:00.688 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125228.log 2026-03-31T21:28:00.688 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125225.log.gz 2026-03-31T21:28:00.688 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125224.log 2026-03-31T21:28:00.689 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125228.log.gz 2026-03-31T21:28:00.689 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125227.log 2026-03-31T21:28:00.689 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125224.log.gz 2026-03-31T21:28:00.689 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125392.log 2026-03-31T21:28:00.690 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125227.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125227.log.gz 2026-03-31T21:28:00.690 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125395.log 2026-03-31T21:28:00.690 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125392.log.gz 2026-03-31T21:28:00.691 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125397.log 2026-03-31T21:28:00.691 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125395.log.gz 2026-03-31T21:28:00.691 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125482.log 2026-03-31T21:28:00.692 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125397.log.gz 2026-03-31T21:28:00.692 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125539.log 2026-03-31T21:28:00.692 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125482.log.gz 2026-03-31T21:28:00.693 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125592.log 2026-03-31T21:28:00.693 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125539.log.gz 2026-03-31T21:28:00.693 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125611.log 2026-03-31T21:28:00.694 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125592.log.gz 2026-03-31T21:28:00.694 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125641.log 2026-03-31T21:28:00.694 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125611.log.gz 2026-03-31T21:28:00.695 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125735.log 2026-03-31T21:28:00.695 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125641.log.gz 2026-03-31T21:28:00.695 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125864.log 2026-03-31T21:28:00.696 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125735.log.gz 2026-03-31T21:28:00.696 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125861.log 2026-03-31T21:28:00.696 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125864.log.gz 2026-03-31T21:28:00.697 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125865.log 2026-03-31T21:28:00.697 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125861.log.gz 2026-03-31T21:28:00.697 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125863.log 2026-03-31T21:28:00.698 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125865.log.gz 2026-03-31T21:28:00.698 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126033.log 2026-03-31T21:28:00.698 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.125863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.125863.log.gz 2026-03-31T21:28:00.699 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126034.log 2026-03-31T21:28:00.699 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126033.log.gz 2026-03-31T21:28:00.699 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126032.log 2026-03-31T21:28:00.700 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126034.log.gz 2026-03-31T21:28:00.700 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126106.log 2026-03-31T21:28:00.700 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126032.log.gz 2026-03-31T21:28:00.701 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126176.log 2026-03-31T21:28:00.701 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126106.log.gz 2026-03-31T21:28:00.701 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126236.log 2026-03-31T21:28:00.702 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126176.log.gz 2026-03-31T21:28:00.702 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126240.log 2026-03-31T21:28:00.702 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126236.log.gz 2026-03-31T21:28:00.703 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126281.log 2026-03-31T21:28:00.703 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126240.log.gz 2026-03-31T21:28:00.703 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126372.log 2026-03-31T21:28:00.704 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126281.log.gz 2026-03-31T21:28:00.704 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126501.log 2026-03-31T21:28:00.704 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126372.log.gz 2026-03-31T21:28:00.705 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126502.log 2026-03-31T21:28:00.705 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126501.log.gz 2026-03-31T21:28:00.705 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126500.log 2026-03-31T21:28:00.706 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126502.log.gz 2026-03-31T21:28:00.706 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126498.log 2026-03-31T21:28:00.706 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126500.log.gz 2026-03-31T21:28:00.707 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126667.log 2026-03-31T21:28:00.707 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126498.log.gz 2026-03-31T21:28:00.707 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126669.log 2026-03-31T21:28:00.708 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126667.log.gz 2026-03-31T21:28:00.708 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126668.log 2026-03-31T21:28:00.708 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126669.log.gz 2026-03-31T21:28:00.709 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126758.log 2026-03-31T21:28:00.709 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126668.log.gz 2026-03-31T21:28:00.709 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126866.log 2026-03-31T21:28:00.710 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126758.log.gz 2026-03-31T21:28:00.710 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126867.log 2026-03-31T21:28:00.710 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126866.log.gz 2026-03-31T21:28:00.711 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126865.log 2026-03-31T21:28:00.711 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126867.log.gz 2026-03-31T21:28:00.711 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126954.log 2026-03-31T21:28:00.712 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126865.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126865.log.gz 2026-03-31T21:28:00.712 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127009.log 2026-03-31T21:28:00.712 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.126954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.126954.log.gz 2026-03-31T21:28:00.713 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127064.log 2026-03-31T21:28:00.713 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127009.log.gz 2026-03-31T21:28:00.713 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127063.log 2026-03-31T21:28:00.714 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127064.log.gz 2026-03-31T21:28:00.714 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127150.log 2026-03-31T21:28:00.714 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127063.log.gz 2026-03-31T21:28:00.715 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127205.log 2026-03-31T21:28:00.715 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127150.log.gz 2026-03-31T21:28:00.715 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127334.log 2026-03-31T21:28:00.716 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127205.log.gz 2026-03-31T21:28:00.716 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127333.log 2026-03-31T21:28:00.716 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127334.log.gz 2026-03-31T21:28:00.717 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127335.log 2026-03-31T21:28:00.717 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127333.log.gz 2026-03-31T21:28:00.717 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127332.log 2026-03-31T21:28:00.718 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127335.log.gz 2026-03-31T21:28:00.718 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127501.log 2026-03-31T21:28:00.718 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127332.log.gz 2026-03-31T21:28:00.719 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127500.log 2026-03-31T21:28:00.719 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127501.log.gz 2026-03-31T21:28:00.719 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127504.log 2026-03-31T21:28:00.719 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127500.log.gz 2026-03-31T21:28:00.720 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127541.log 2026-03-31T21:28:00.720 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127504.log.gz 2026-03-31T21:28:00.720 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127646.log 2026-03-31T21:28:00.721 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127541.log.gz 2026-03-31T21:28:00.721 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127700.log 2026-03-31T21:28:00.722 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127646.log.gz 2026-03-31T21:28:00.722 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127710.log 2026-03-31T21:28:00.722 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127700.log.gz 2026-03-31T21:28:00.722 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127751.log 2026-03-31T21:28:00.723 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127710.log.gz 2026-03-31T21:28:00.723 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127842.log 2026-03-31T21:28:00.724 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127751.log.gz 2026-03-31T21:28:00.724 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127869.log 2026-03-31T21:28:00.724 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127842.log.gz 2026-03-31T21:28:00.724 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127940.log 2026-03-31T21:28:00.725 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127869.log.gz 2026-03-31T21:28:00.725 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128067.log 2026-03-31T21:28:00.725 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.127940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.127940.log.gz 2026-03-31T21:28:00.726 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128068.log 2026-03-31T21:28:00.726 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128067.log.gz 2026-03-31T21:28:00.726 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128065.log 2026-03-31T21:28:00.727 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128068.log.gz 2026-03-31T21:28:00.727 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128070.log 2026-03-31T21:28:00.728 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128065.log.gz 2026-03-31T21:28:00.728 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128185.log 2026-03-31T21:28:00.728 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128070.log.gz 2026-03-31T21:28:00.729 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128241.log 2026-03-31T21:28:00.729 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128185.log.gz 2026-03-31T21:28:00.729 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128240.log 2026-03-31T21:28:00.730 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128241.log.gz 2026-03-31T21:28:00.730 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128307.log 2026-03-31T21:28:00.730 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128240.log.gz 2026-03-31T21:28:00.731 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128381.log 2026-03-31T21:28:00.731 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128307.log.gz 2026-03-31T21:28:00.731 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128410.log 2026-03-31T21:28:00.732 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128381.log.gz 2026-03-31T21:28:00.732 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128456.log 2026-03-31T21:28:00.732 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128410.log.gz 2026-03-31T21:28:00.733 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128503.log 2026-03-31T21:28:00.733 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128456.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.128456.log.gz 2026-03-31T21:28:00.734 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128577.log 2026-03-31T21:28:00.734 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128503.log.gz 2026-03-31T21:28:00.734 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128704.log 2026-03-31T21:28:00.735 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128577.log.gz 2026-03-31T21:28:00.735 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128702.log 2026-03-31T21:28:00.735 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128704.log.gz 2026-03-31T21:28:00.736 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128703.log 2026-03-31T21:28:00.736 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128702.log.gz 2026-03-31T21:28:00.736 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128707.log 2026-03-31T21:28:00.737 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128703.log.gz 2026-03-31T21:28:00.737 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128875.log 2026-03-31T21:28:00.737 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128707.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128707.log.gz 2026-03-31T21:28:00.738 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128876.log 2026-03-31T21:28:00.738 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128875.log.gz 2026-03-31T21:28:00.738 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128872.log 2026-03-31T21:28:00.739 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128876.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128876.log.gz 2026-03-31T21:28:00.739 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128935.log 2026-03-31T21:28:00.740 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128872.log.gz 2026-03-31T21:28:00.740 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129018.log 2026-03-31T21:28:00.740 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.128935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.128935.log.gz 2026-03-31T21:28:00.741 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129072.log 2026-03-31T21:28:00.741 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129018.log.gz 2026-03-31T21:28:00.741 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129071.log 2026-03-31T21:28:00.742 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129072.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129072.log.gz 2026-03-31T21:28:00.742 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129149.log 2026-03-31T21:28:00.742 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129071.log.gz 2026-03-31T21:28:00.743 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129214.log 2026-03-31T21:28:00.743 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129149.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129149.log.gz 2026-03-31T21:28:00.743 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129340.log 2026-03-31T21:28:00.744 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129214.log.gz 2026-03-31T21:28:00.744 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129341.log 2026-03-31T21:28:00.744 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129340.log.gz 2026-03-31T21:28:00.745 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129344.log 2026-03-31T21:28:00.745 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129341.log.gz 2026-03-31T21:28:00.746 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129343.log 2026-03-31T21:28:00.746 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129344.log.gz 2026-03-31T21:28:00.746 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129488.log 2026-03-31T21:28:00.747 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129343.log.gz 2026-03-31T21:28:00.747 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129510.log 2026-03-31T21:28:00.747 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129488.log.gz 2026-03-31T21:28:00.748 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129513.log 2026-03-31T21:28:00.748 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129510.log.gz 2026-03-31T21:28:00.748 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129561.log 2026-03-31T21:28:00.749 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129513.log.gz 2026-03-31T21:28:00.749 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129655.log 2026-03-31T21:28:00.749 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129561.log.gz 2026-03-31T21:28:00.750 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129735.log 2026-03-31T21:28:00.750 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129655.log.gz 2026-03-31T21:28:00.750 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129734.log 2026-03-31T21:28:00.751 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129735.log.gz 2026-03-31T21:28:00.751 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129736.log 2026-03-31T21:28:00.752 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129734.log.gz 2026-03-31T21:28:00.752 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129851.log 2026-03-31T21:28:00.752 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129736.log.gz 2026-03-31T21:28:00.753 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129900.log 2026-03-31T21:28:00.753 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129851.log.gz 2026-03-31T21:28:00.753 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130029.log 2026-03-31T21:28:00.754 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.129900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.129900.log.gz 2026-03-31T21:28:00.754 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130030.log 2026-03-31T21:28:00.754 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130029.log.gz 2026-03-31T21:28:00.755 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130027.log 2026-03-31T21:28:00.755 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130030.log.gz 2026-03-31T21:28:00.755 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130028.log 2026-03-31T21:28:00.756 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130027.log.gz 2026-03-31T21:28:00.756 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130222.log 2026-03-31T21:28:00.756 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130028.log.gz 2026-03-31T21:28:00.757 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130221.log 2026-03-31T21:28:00.757 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130222.log.gz 2026-03-31T21:28:00.757 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130224.log 2026-03-31T21:28:00.758 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130221.log.gz 2026-03-31T21:28:00.758 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130226.log 2026-03-31T21:28:00.758 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130224.log.gz 2026-03-31T21:28:00.759 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130341.log 2026-03-31T21:28:00.759 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130226.log.gz 2026-03-31T21:28:00.759 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130417.log 2026-03-31T21:28:00.760 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130341.log.gz 2026-03-31T21:28:00.760 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130418.log 2026-03-31T21:28:00.760 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130417.log.gz 2026-03-31T21:28:00.761 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130422.log 2026-03-31T21:28:00.761 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130418.log.gz 2026-03-31T21:28:00.761 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130537.log 2026-03-31T21:28:00.762 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130422.log.gz 2026-03-31T21:28:00.762 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130666.log 2026-03-31T21:28:00.762 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130537.log.gz 2026-03-31T21:28:00.763 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130667.log 2026-03-31T21:28:00.763 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130666.log.gz 2026-03-31T21:28:00.763 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130665.log 2026-03-31T21:28:00.764 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130667.log.gz 2026-03-31T21:28:00.764 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130664.log 2026-03-31T21:28:00.765 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130665.log.gz 2026-03-31T21:28:00.765 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130770.log 2026-03-31T21:28:00.765 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130664.log: 29.4% -- replaced with /var/log/ceph/ceph-client.admin.130664.log.gz 2026-03-31T21:28:00.766 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130834.log 2026-03-31T21:28:00.766 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130770.log.gz 2026-03-31T21:28:00.766 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130833.log 2026-03-31T21:28:00.767 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130834.log.gz 2026-03-31T21:28:00.767 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130925.log 2026-03-31T21:28:00.767 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130833.log.gz 2026-03-31T21:28:00.768 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130978.log 2026-03-31T21:28:00.768 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130925.log.gz 2026-03-31T21:28:00.768 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131033.log 2026-03-31T21:28:00.769 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.130978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.130978.log.gz 2026-03-31T21:28:00.769 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131034.log 2026-03-31T21:28:00.769 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131033.log.gz 2026-03-31T21:28:00.770 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131121.log 2026-03-31T21:28:00.770 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131034.log.gz 2026-03-31T21:28:00.770 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131174.log 2026-03-31T21:28:00.771 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131121.log.gz 2026-03-31T21:28:00.771 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131301.log 2026-03-31T21:28:00.771 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131174.log.gz 2026-03-31T21:28:00.772 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131302.log 2026-03-31T21:28:00.772 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131301.log.gz 2026-03-31T21:28:00.772 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131304.log 2026-03-31T21:28:00.773 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131302.log.gz 2026-03-31T21:28:00.773 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131303.log 2026-03-31T21:28:00.773 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131304.log.gz 2026-03-31T21:28:00.774 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131481.log 2026-03-31T21:28:00.774 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131303.log.gz 2026-03-31T21:28:00.774 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131500.log 2026-03-31T21:28:00.775 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131481.log.gz 2026-03-31T21:28:00.775 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131496.log 2026-03-31T21:28:00.775 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131500.log.gz 2026-03-31T21:28:00.776 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131497.log 2026-03-31T21:28:00.776 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131496.log.gz 2026-03-31T21:28:00.776 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131615.log 2026-03-31T21:28:00.777 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131497.log.gz 2026-03-31T21:28:00.777 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131696.log 2026-03-31T21:28:00.777 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131615.log.gz 2026-03-31T21:28:00.778 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131695.log 2026-03-31T21:28:00.778 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131696.log.gz 2026-03-31T21:28:00.778 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131698.log 2026-03-31T21:28:00.779 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131695.log.gz 2026-03-31T21:28:00.779 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131811.log 2026-03-31T21:28:00.779 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131698.log.gz 2026-03-31T21:28:00.780 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131890.log 2026-03-31T21:28:00.780 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131811.log.gz 2026-03-31T21:28:00.781 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131892.log 2026-03-31T21:28:00.781 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131890.log.gz 2026-03-31T21:28:00.781 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.131893.log 2026-03-31T21:28:00.782 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131892.log.gz 2026-03-31T21:28:00.782 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132007.log 2026-03-31T21:28:00.782 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.131893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.131893.log.gz 2026-03-31T21:28:00.783 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132136.log 2026-03-31T21:28:00.783 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132007.log.gz 2026-03-31T21:28:00.783 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132135.log 2026-03-31T21:28:00.784 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132136.log.gz 2026-03-31T21:28:00.784 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132129.log 2026-03-31T21:28:00.784 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132135.log.gz 2026-03-31T21:28:00.785 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132137.log 2026-03-31T21:28:00.785 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132129.log.gz 2026-03-31T21:28:00.785 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132286.log 2026-03-31T21:28:00.786 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132137.log.gz 2026-03-31T21:28:00.786 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132333.log 2026-03-31T21:28:00.786 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132286.log.gz 2026-03-31T21:28:00.787 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132331.log 2026-03-31T21:28:00.787 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132333.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132333.log.gz 2026-03-31T21:28:00.787 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132332.log 2026-03-31T21:28:00.788 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132331.log.gz 2026-03-31T21:28:00.788 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132448.log 2026-03-31T21:28:00.788 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132332.log.gz 2026-03-31T21:28:00.789 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132525.log 2026-03-31T21:28:00.789 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132448.log.gz 2026-03-31T21:28:00.789 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132524.log 2026-03-31T21:28:00.790 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132525.log.gz 2026-03-31T21:28:00.790 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132530.log 2026-03-31T21:28:00.790 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132524.log.gz 2026-03-31T21:28:00.791 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132644.log 2026-03-31T21:28:00.791 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132530.log.gz 2026-03-31T21:28:00.791 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132773.log 2026-03-31T21:28:00.792 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132644.log.gz 2026-03-31T21:28:00.792 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132771.log 2026-03-31T21:28:00.792 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132773.log.gz 2026-03-31T21:28:00.793 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132774.log 2026-03-31T21:28:00.793 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132771.log.gz 2026-03-31T21:28:00.793 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132772.log 2026-03-31T21:28:00.794 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132774.log.gz 2026-03-31T21:28:00.794 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132956.log 2026-03-31T21:28:00.794 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132772.log.gz 2026-03-31T21:28:00.795 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132970.log 2026-03-31T21:28:00.795 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132956.log.gz 2026-03-31T21:28:00.796 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132969.log 2026-03-31T21:28:00.796 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132970.log.gz 2026-03-31T21:28:00.796 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132967.log 2026-03-31T21:28:00.797 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132969.log.gz 2026-03-31T21:28:00.797 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133085.log 2026-03-31T21:28:00.798 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.132967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.132967.log.gz 2026-03-31T21:28:00.798 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133162.log 2026-03-31T21:28:00.798 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133085.log.gz 2026-03-31T21:28:00.799 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133165.log 2026-03-31T21:28:00.799 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133162.log.gz 2026-03-31T21:28:00.799 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133166.log 2026-03-31T21:28:00.800 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133165.log.gz 2026-03-31T21:28:00.800 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133281.log 2026-03-31T21:28:00.800 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133166.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133166.log.gz 2026-03-31T21:28:00.801 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133409.log 2026-03-31T21:28:00.801 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133281.log.gz 2026-03-31T21:28:00.801 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133410.log 2026-03-31T21:28:00.802 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133409.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133409.log.gz 2026-03-31T21:28:00.802 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133406.log 2026-03-31T21:28:00.802 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133410.log.gz 2026-03-31T21:28:00.803 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133411.log 2026-03-31T21:28:00.803 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133406.log.gz 2026-03-31T21:28:00.803 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133553.log 2026-03-31T21:28:00.804 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133411.log.gz 2026-03-31T21:28:00.804 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133552.log 2026-03-31T21:28:00.804 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133553.log.gz 2026-03-31T21:28:00.805 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133644.log 2026-03-31T21:28:00.805 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133552.log.gz 2026-03-31T21:28:00.805 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133647.log 2026-03-31T21:28:00.806 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133644.log.gz 2026-03-31T21:28:00.806 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133722.log 2026-03-31T21:28:00.806 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133647.log.gz 2026-03-31T21:28:00.807 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133749.log 2026-03-31T21:28:00.807 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133722.log.gz 2026-03-31T21:28:00.807 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133823.log 2026-03-31T21:28:00.808 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133749.log.gz 2026-03-31T21:28:00.808 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133830.log 2026-03-31T21:28:00.808 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133823.log.gz 2026-03-31T21:28:00.809 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133918.log 2026-03-31T21:28:00.809 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133830.log.gz 2026-03-31T21:28:00.809 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133965.log 2026-03-31T21:28:00.810 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133918.log.gz 2026-03-31T21:28:00.810 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134018.log 2026-03-31T21:28:00.810 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.133965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.133965.log.gz 2026-03-31T21:28:00.811 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134021.log 2026-03-31T21:28:00.811 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134018.log.gz 2026-03-31T21:28:00.811 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134114.log 2026-03-31T21:28:00.812 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134021.log.gz 2026-03-31T21:28:00.812 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134243.log 2026-03-31T21:28:00.812 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134114.log.gz 2026-03-31T21:28:00.813 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134241.log 2026-03-31T21:28:00.813 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134243.log.gz 2026-03-31T21:28:00.813 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134242.log 2026-03-31T21:28:00.814 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134241.log.gz 2026-03-31T21:28:00.814 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134244.log 2026-03-31T21:28:00.814 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134242.log: 27.2% -- replaced with /var/log/ceph/ceph-client.admin.134242.log.gz 2026-03-31T21:28:00.815 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134345.log 2026-03-31T21:28:00.815 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134244.log.gz 2026-03-31T21:28:00.815 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134381.log 2026-03-31T21:28:00.816 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134345.log.gz 2026-03-31T21:28:00.816 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134455.log 2026-03-31T21:28:00.816 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134381.log.gz 2026-03-31T21:28:00.817 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134502.log 2026-03-31T21:28:00.817 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134455.log.gz 2026-03-31T21:28:00.817 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134555.log 2026-03-31T21:28:00.818 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134502.log.gz 2026-03-31T21:28:00.818 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134600.log 2026-03-31T21:28:00.818 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134555.log.gz 2026-03-31T21:28:00.819 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134651.log 2026-03-31T21:28:00.819 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134600.log.gz 2026-03-31T21:28:00.819 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134681.log 2026-03-31T21:28:00.820 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134651.log.gz 2026-03-31T21:28:00.820 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134751.log 2026-03-31T21:28:00.820 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134681.log.gz 2026-03-31T21:28:00.821 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134881.log 2026-03-31T21:28:00.821 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134751.log.gz 2026-03-31T21:28:00.821 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134880.log 2026-03-31T21:28:00.822 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134881.log.gz 2026-03-31T21:28:00.822 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134878.log 2026-03-31T21:28:00.822 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134880.log: 28.3% -- replaced with /var/log/ceph/ceph-client.admin.134880.log.gz 2026-03-31T21:28:00.823 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134879.log 2026-03-31T21:28:00.823 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134878.log.gz 2026-03-31T21:28:00.823 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134982.log 2026-03-31T21:28:00.824 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134879.log.gz 2026-03-31T21:28:00.824 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135021.log 2026-03-31T21:28:00.824 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.134982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.134982.log.gz 2026-03-31T21:28:00.825 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135113.log 2026-03-31T21:28:00.825 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135021.log.gz 2026-03-31T21:28:00.825 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135122.log 2026-03-31T21:28:00.826 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135113.log.gz 2026-03-31T21:28:00.826 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135195.log 2026-03-31T21:28:00.826 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135122.log.gz 2026-03-31T21:28:00.827 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135222.log 2026-03-31T21:28:00.827 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135195.log.gz 2026-03-31T21:28:00.827 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135293.log 2026-03-31T21:28:00.828 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135222.log.gz 2026-03-31T21:28:00.828 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135320.log 2026-03-31T21:28:00.828 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135293.log.gz 2026-03-31T21:28:00.829 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135391.log 2026-03-31T21:28:00.829 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135320.log.gz 2026-03-31T21:28:00.829 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135516.log 2026-03-31T21:28:00.830 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135391.log.gz 2026-03-31T21:28:00.830 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135518.log 2026-03-31T21:28:00.830 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135516.log.gz 2026-03-31T21:28:00.831 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135519.log 2026-03-31T21:28:00.831 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135518.log.gz 2026-03-31T21:28:00.831 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135522.log 2026-03-31T21:28:00.832 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135519.log.gz 2026-03-31T21:28:00.832 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135699.log 2026-03-31T21:28:00.832 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135522.log.gz 2026-03-31T21:28:00.833 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135713.log 2026-03-31T21:28:00.833 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135699.log.gz 2026-03-31T21:28:00.833 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135715.log 2026-03-31T21:28:00.834 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135713.log.gz 2026-03-31T21:28:00.834 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135718.log 2026-03-31T21:28:00.834 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135715.log.gz 2026-03-31T21:28:00.835 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135833.log 2026-03-31T21:28:00.835 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135718.log.gz 2026-03-31T21:28:00.835 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135912.log 2026-03-31T21:28:00.836 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135833.log.gz 2026-03-31T21:28:00.836 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135913.log 2026-03-31T21:28:00.836 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135912.log.gz 2026-03-31T21:28:00.837 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135915.log 2026-03-31T21:28:00.837 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135913.log.gz 2026-03-31T21:28:00.837 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136029.log 2026-03-31T21:28:00.838 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.135915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.135915.log.gz 2026-03-31T21:28:00.838 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136110.log 2026-03-31T21:28:00.838 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136029.log.gz 2026-03-31T21:28:00.839 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136107.log 2026-03-31T21:28:00.839 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136110.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136110.log.gz 2026-03-31T21:28:00.839 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136111.log 2026-03-31T21:28:00.840 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136107.log.gz 2026-03-31T21:28:00.840 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136225.log 2026-03-31T21:28:00.840 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136111.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136111.log.gz 2026-03-31T21:28:00.841 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136353.log 2026-03-31T21:28:00.841 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136225.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136225.log.gz 2026-03-31T21:28:00.842 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136354.log 2026-03-31T21:28:00.842 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136353.log.gz 2026-03-31T21:28:00.842 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136355.log 2026-03-31T21:28:00.843 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136354.log.gz 2026-03-31T21:28:00.843 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136351.log 2026-03-31T21:28:00.844 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136355.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136355.log.gz 2026-03-31T21:28:00.844 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136519.log 2026-03-31T21:28:00.844 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136351.log.gz 2026-03-31T21:28:00.845 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136549.log 2026-03-31T21:28:00.845 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136519.log.gz 2026-03-31T21:28:00.845 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136550.log 2026-03-31T21:28:00.846 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136549.log.gz 2026-03-31T21:28:00.846 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136551.log 2026-03-31T21:28:00.846 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136550.log.gz 2026-03-31T21:28:00.847 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136678.log 2026-03-31T21:28:00.847 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136551.log.gz 2026-03-31T21:28:00.847 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136757.log 2026-03-31T21:28:00.848 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136678.log.gz 2026-03-31T21:28:00.848 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136760.log 2026-03-31T21:28:00.848 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136757.log.gz 2026-03-31T21:28:00.849 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136759.log 2026-03-31T21:28:00.849 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136760.log.gz 2026-03-31T21:28:00.849 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136874.log 2026-03-31T21:28:00.850 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136759.log.gz 2026-03-31T21:28:00.850 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136928.log 2026-03-31T21:28:00.850 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136874.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136874.log.gz 2026-03-31T21:28:00.851 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136929.log 2026-03-31T21:28:00.851 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136928.log.gz 2026-03-31T21:28:00.851 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136995.log 2026-03-31T21:28:00.852 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136929.log.gz 2026-03-31T21:28:00.852 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137070.log 2026-03-31T21:28:00.852 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.136995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.136995.log.gz 2026-03-31T21:28:00.852 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137200.log 2026-03-31T21:28:00.853 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137070.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137070.log.gz 2026-03-31T21:28:00.853 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137198.log 2026-03-31T21:28:00.854 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137200.log.gz 2026-03-31T21:28:00.854 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137199.log 2026-03-31T21:28:00.854 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137198.log.gz 2026-03-31T21:28:00.854 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137197.log 2026-03-31T21:28:00.855 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137199.log.gz 2026-03-31T21:28:00.855 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137384.log 2026-03-31T21:28:00.856 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137197.log.gz 2026-03-31T21:28:00.856 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137396.log 2026-03-31T21:28:00.856 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137384.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137384.log.gz 2026-03-31T21:28:00.856 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137394.log 2026-03-31T21:28:00.857 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137396.log.gz 2026-03-31T21:28:00.857 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137395.log 2026-03-31T21:28:00.857 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137394.log.gz 2026-03-31T21:28:00.858 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137538.log 2026-03-31T21:28:00.858 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137395.log.gz 2026-03-31T21:28:00.858 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137537.log 2026-03-31T21:28:00.859 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137538.log.gz 2026-03-31T21:28:00.859 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137630.log 2026-03-31T21:28:00.859 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137537.log.gz 2026-03-31T21:28:00.860 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137616.log 2026-03-31T21:28:00.860 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137630.log.gz 2026-03-31T21:28:00.860 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137708.log 2026-03-31T21:28:00.861 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137616.log.gz 2026-03-31T21:28:00.861 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137735.log 2026-03-31T21:28:00.861 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137708.log.gz 2026-03-31T21:28:00.862 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137805.log 2026-03-31T21:28:00.862 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137735.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.137735.log.gz 2026-03-31T21:28:00.862 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137819.log 2026-03-31T21:28:00.863 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137805.log.gz 2026-03-31T21:28:00.863 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137904.log 2026-03-31T21:28:00.863 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137819.log.gz 2026-03-31T21:28:00.864 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138031.log 2026-03-31T21:28:00.864 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.137904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.137904.log.gz 2026-03-31T21:28:00.864 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138033.log 2026-03-31T21:28:00.865 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138031.log.gz 2026-03-31T21:28:00.865 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138034.log 2026-03-31T21:28:00.865 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138033.log.gz 2026-03-31T21:28:00.866 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138029.log 2026-03-31T21:28:00.866 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138034.log.gz 2026-03-31T21:28:00.867 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138218.log 2026-03-31T21:28:00.867 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138029.log.gz 2026-03-31T21:28:00.867 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138229.log 2026-03-31T21:28:00.868 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138218.log.gz 2026-03-31T21:28:00.868 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138228.log 2026-03-31T21:28:00.868 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138229.log.gz 2026-03-31T21:28:00.869 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138230.log 2026-03-31T21:28:00.869 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138228.log.gz 2026-03-31T21:28:00.869 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138345.log 2026-03-31T21:28:00.869 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138230.log.gz 2026-03-31T21:28:00.870 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138427.log 2026-03-31T21:28:00.870 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138345.log.gz 2026-03-31T21:28:00.870 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138426.log 2026-03-31T21:28:00.871 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138427.log.gz 2026-03-31T21:28:00.871 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138428.log 2026-03-31T21:28:00.871 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138426.log.gz 2026-03-31T21:28:00.872 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138541.log 2026-03-31T21:28:00.872 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138428.log.gz 2026-03-31T21:28:00.873 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138669.log 2026-03-31T21:28:00.873 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138541.log.gz 2026-03-31T21:28:00.873 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138670.log 2026-03-31T21:28:00.874 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138669.log.gz 2026-03-31T21:28:00.874 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138671.log 2026-03-31T21:28:00.874 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138670.log.gz 2026-03-31T21:28:00.875 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138668.log 2026-03-31T21:28:00.875 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138671.log.gz 2026-03-31T21:28:00.875 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138852.log 2026-03-31T21:28:00.876 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138668.log.gz 2026-03-31T21:28:00.876 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138866.log 2026-03-31T21:28:00.876 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138852.log.gz 2026-03-31T21:28:00.876 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138867.log 2026-03-31T21:28:00.877 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138866.log.gz 2026-03-31T21:28:00.877 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138864.log 2026-03-31T21:28:00.877 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138867.log.gz 2026-03-31T21:28:00.878 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139008.log 2026-03-31T21:28:00.878 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.138864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.138864.log.gz 2026-03-31T21:28:00.878 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139009.log 2026-03-31T21:28:00.879 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139008.log.gz 2026-03-31T21:28:00.879 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139063.log 2026-03-31T21:28:00.879 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139009.log.gz 2026-03-31T21:28:00.880 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139065.log 2026-03-31T21:28:00.880 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139063.log.gz 2026-03-31T21:28:00.880 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139178.log 2026-03-31T21:28:00.881 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139065.log.gz 2026-03-31T21:28:00.881 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139304.log 2026-03-31T21:28:00.881 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139178.log.gz 2026-03-31T21:28:00.882 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139307.log 2026-03-31T21:28:00.882 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139304.log.gz 2026-03-31T21:28:00.882 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139305.log 2026-03-31T21:28:00.883 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139307.log.gz 2026-03-31T21:28:00.883 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139308.log 2026-03-31T21:28:00.883 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139305.log.gz 2026-03-31T21:28:00.884 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139492.log 2026-03-31T21:28:00.884 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139308.log.gz 2026-03-31T21:28:00.885 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139502.log 2026-03-31T21:28:00.885 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139492.log.gz 2026-03-31T21:28:00.885 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139504.log 2026-03-31T21:28:00.886 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139502.log.gz 2026-03-31T21:28:00.886 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139503.log 2026-03-31T21:28:00.887 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139504.log.gz 2026-03-31T21:28:00.887 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139619.log 2026-03-31T21:28:00.887 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139503.log.gz 2026-03-31T21:28:00.888 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139700.log 2026-03-31T21:28:00.888 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139619.log.gz 2026-03-31T21:28:00.888 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139698.log 2026-03-31T21:28:00.889 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139700.log.gz 2026-03-31T21:28:00.889 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139699.log 2026-03-31T21:28:00.889 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139698.log.gz 2026-03-31T21:28:00.890 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139815.log 2026-03-31T21:28:00.890 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139699.log.gz 2026-03-31T21:28:00.890 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139941.log 2026-03-31T21:28:00.891 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139815.log.gz 2026-03-31T21:28:00.891 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139945.log 2026-03-31T21:28:00.892 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139941.log.gz 2026-03-31T21:28:00.892 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139943.log 2026-03-31T21:28:00.892 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139945.log.gz 2026-03-31T21:28:00.892 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139940.log 2026-03-31T21:28:00.893 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139943.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139943.log.gz 2026-03-31T21:28:00.893 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140139.log 2026-03-31T21:28:00.893 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.139940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.139940.log.gz 2026-03-31T21:28:00.894 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140137.log 2026-03-31T21:28:00.894 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140139.log: 30.5% -- replaced with /var/log/ceph/ceph-client.admin.140139.log.gz 2026-03-31T21:28:00.894 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140141.log 2026-03-31T21:28:00.895 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140137.log.gz 2026-03-31T21:28:00.895 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140129.log 2026-03-31T21:28:00.895 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140141.log.gz 2026-03-31T21:28:00.896 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140256.log 2026-03-31T21:28:00.896 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140129.log.gz 2026-03-31T21:28:00.896 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140310.log 2026-03-31T21:28:00.897 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140256.log.gz 2026-03-31T21:28:00.897 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140311.log 2026-03-31T21:28:00.897 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140310.log.gz 2026-03-31T21:28:00.898 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140399.log 2026-03-31T21:28:00.898 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140311.log.gz 2026-03-31T21:28:00.898 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140452.log 2026-03-31T21:28:00.899 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140399.log.gz 2026-03-31T21:28:00.899 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140581.log 2026-03-31T21:28:00.899 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140452.log.gz 2026-03-31T21:28:00.900 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140582.log 2026-03-31T21:28:00.900 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140581.log.gz 2026-03-31T21:28:00.901 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140578.log 2026-03-31T21:28:00.901 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140582.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140582.log.gz 2026-03-31T21:28:00.901 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140577.log 2026-03-31T21:28:00.902 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140578.log.gz 2026-03-31T21:28:00.902 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140749.log 2026-03-31T21:28:00.902 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140577.log.gz 2026-03-31T21:28:00.903 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140747.log 2026-03-31T21:28:00.903 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140749.log.gz 2026-03-31T21:28:00.903 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140748.log 2026-03-31T21:28:00.904 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140747.log.gz 2026-03-31T21:28:00.904 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140844.log 2026-03-31T21:28:00.904 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140748.log.gz 2026-03-31T21:28:00.905 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140893.log 2026-03-31T21:28:00.905 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140844.log.gz 2026-03-31T21:28:00.905 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140947.log 2026-03-31T21:28:00.906 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140893.log.gz 2026-03-31T21:28:00.906 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140946.log 2026-03-31T21:28:00.906 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140947.log.gz 2026-03-31T21:28:00.907 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141036.log 2026-03-31T21:28:00.907 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.140946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.140946.log.gz 2026-03-31T21:28:00.907 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141089.log 2026-03-31T21:28:00.908 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141036.log.gz 2026-03-31T21:28:00.908 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141117.log 2026-03-31T21:28:00.908 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141089.log.gz 2026-03-31T21:28:00.909 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141163.log 2026-03-31T21:28:00.909 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141117.log.gz 2026-03-31T21:28:00.909 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141236.log 2026-03-31T21:28:00.910 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141163.log.gz 2026-03-31T21:28:00.910 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141362.log 2026-03-31T21:28:00.910 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141236.log.gz 2026-03-31T21:28:00.911 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141364.log 2026-03-31T21:28:00.911 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141362.log.gz 2026-03-31T21:28:00.911 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141365.log 2026-03-31T21:28:00.912 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141364.log.gz 2026-03-31T21:28:00.912 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141366.log 2026-03-31T21:28:00.912 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141365.log.gz 2026-03-31T21:28:00.913 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141559.log 2026-03-31T21:28:00.913 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141366.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141366.log.gz 2026-03-31T21:28:00.913 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141560.log 2026-03-31T21:28:00.914 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141559.log.gz 2026-03-31T21:28:00.914 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141562.log 2026-03-31T21:28:00.914 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141560.log.gz 2026-03-31T21:28:00.915 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141561.log 2026-03-31T21:28:00.915 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141562.log.gz 2026-03-31T21:28:00.915 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141677.log 2026-03-31T21:28:00.916 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141561.log.gz 2026-03-31T21:28:00.916 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141755.log 2026-03-31T21:28:00.916 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141677.log.gz 2026-03-31T21:28:00.917 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141757.log 2026-03-31T21:28:00.917 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141755.log.gz 2026-03-31T21:28:00.917 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141759.log 2026-03-31T21:28:00.918 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141757.log.gz 2026-03-31T21:28:00.918 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141873.log 2026-03-31T21:28:00.918 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141759.log.gz 2026-03-31T21:28:00.919 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142001.log 2026-03-31T21:28:00.919 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141873.log.gz 2026-03-31T21:28:00.919 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141999.log 2026-03-31T21:28:00.920 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142001.log.gz 2026-03-31T21:28:00.920 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142003.log 2026-03-31T21:28:00.920 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.141999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.141999.log.gz 2026-03-31T21:28:00.921 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142002.log 2026-03-31T21:28:00.921 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142003.log.gz 2026-03-31T21:28:00.921 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142168.log 2026-03-31T21:28:00.922 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142002.log.gz 2026-03-31T21:28:00.922 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142169.log 2026-03-31T21:28:00.922 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142168.log.gz 2026-03-31T21:28:00.923 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142172.log 2026-03-31T21:28:00.923 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142169.log.gz 2026-03-31T21:28:00.923 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142214.log 2026-03-31T21:28:00.924 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142172.log.gz 2026-03-31T21:28:00.924 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142314.log 2026-03-31T21:28:00.924 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142214.log.gz 2026-03-31T21:28:00.925 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142368.log 2026-03-31T21:28:00.925 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142314.log.gz 2026-03-31T21:28:00.925 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142370.log 2026-03-31T21:28:00.926 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142368.log.gz 2026-03-31T21:28:00.926 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142448.log 2026-03-31T21:28:00.926 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142370.log.gz 2026-03-31T21:28:00.927 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142510.log 2026-03-31T21:28:00.927 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142448.log.gz 2026-03-31T21:28:00.927 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142638.log 2026-03-31T21:28:00.928 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142510.log.gz 2026-03-31T21:28:00.928 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142640.log 2026-03-31T21:28:00.928 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142638.log.gz 2026-03-31T21:28:00.929 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142639.log 2026-03-31T21:28:00.929 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142640.log.gz 2026-03-31T21:28:00.929 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142637.log 2026-03-31T21:28:00.930 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142639.log.gz 2026-03-31T21:28:00.930 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142780.log 2026-03-31T21:28:00.930 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142637.log.gz 2026-03-31T21:28:00.931 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142782.log 2026-03-31T21:28:00.931 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142780.log.gz 2026-03-31T21:28:00.931 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142855.log 2026-03-31T21:28:00.932 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142782.log.gz 2026-03-31T21:28:00.932 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142856.log 2026-03-31T21:28:00.932 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142855.log.gz 2026-03-31T21:28:00.933 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142951.log 2026-03-31T21:28:00.933 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142856.log: 30.1% -- replaced with /var/log/ceph/ceph-client.admin.142856.log.gz 2026-03-31T21:28:00.933 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143006.log 2026-03-31T21:28:00.934 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.142951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.142951.log.gz 2026-03-31T21:28:00.934 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143007.log 2026-03-31T21:28:00.934 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143006.log.gz 2026-03-31T21:28:00.935 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143098.log 2026-03-31T21:28:00.935 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143007.log.gz 2026-03-31T21:28:00.935 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143147.log 2026-03-31T21:28:00.936 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143098.log.gz 2026-03-31T21:28:00.936 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143187.log 2026-03-31T21:28:00.936 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143147.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143147.log.gz 2026-03-31T21:28:00.937 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143223.log 2026-03-31T21:28:00.937 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143187.log.gz 2026-03-31T21:28:00.937 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143294.log 2026-03-31T21:28:00.938 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143223.log.gz 2026-03-31T21:28:00.938 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143419.log 2026-03-31T21:28:00.938 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143294.log.gz 2026-03-31T21:28:00.939 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143420.log 2026-03-31T21:28:00.939 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143419.log.gz 2026-03-31T21:28:00.939 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143421.log 2026-03-31T21:28:00.940 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143420.log.gz 2026-03-31T21:28:00.940 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143424.log 2026-03-31T21:28:00.940 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143421.log.gz 2026-03-31T21:28:00.941 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143583.log 2026-03-31T21:28:00.941 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143424.log.gz 2026-03-31T21:28:00.941 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143588.log 2026-03-31T21:28:00.942 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143583.log.gz 2026-03-31T21:28:00.942 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143591.log 2026-03-31T21:28:00.942 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143588.log.gz 2026-03-31T21:28:00.943 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143686.log 2026-03-31T21:28:00.943 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143591.log.gz 2026-03-31T21:28:00.943 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143735.log 2026-03-31T21:28:00.944 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143686.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143686.log.gz 2026-03-31T21:28:00.944 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143797.log 2026-03-31T21:28:00.944 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143735.log.gz 2026-03-31T21:28:00.945 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143799.log 2026-03-31T21:28:00.945 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143797.log.gz 2026-03-31T21:28:00.945 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143878.log 2026-03-31T21:28:00.946 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143799.log.gz 2026-03-31T21:28:00.946 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143931.log 2026-03-31T21:28:00.946 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143878.log.gz 2026-03-31T21:28:00.947 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144002.log 2026-03-31T21:28:00.947 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.143931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.143931.log.gz 2026-03-31T21:28:00.947 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144004.log 2026-03-31T21:28:00.948 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144002.log: 29.7% -- replaced with /var/log/ceph/ceph-client.admin.144002.log.gz 2026-03-31T21:28:00.948 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144078.log 2026-03-31T21:28:00.948 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144004.log.gz 2026-03-31T21:28:00.949 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144207.log 2026-03-31T21:28:00.949 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144078.log.gz 2026-03-31T21:28:00.949 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144206.log 2026-03-31T21:28:00.950 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144207.log.gz 2026-03-31T21:28:00.950 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144208.log 2026-03-31T21:28:00.950 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144206.log.gz 2026-03-31T21:28:00.951 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144205.log 2026-03-31T21:28:00.951 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144208.log.gz 2026-03-31T21:28:00.951 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144323.log 2026-03-31T21:28:00.952 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144205.log.gz 2026-03-31T21:28:00.952 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144387.log 2026-03-31T21:28:00.952 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144323.log.gz 2026-03-31T21:28:00.953 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144386.log 2026-03-31T21:28:00.953 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144387.log.gz 2026-03-31T21:28:00.953 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144424.log 2026-03-31T21:28:00.954 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144386.log.gz 2026-03-31T21:28:00.954 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144519.log 2026-03-31T21:28:00.954 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144424.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144424.log.gz 2026-03-31T21:28:00.955 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144550.log 2026-03-31T21:28:00.955 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144519.log.gz 2026-03-31T21:28:00.955 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144618.log 2026-03-31T21:28:00.956 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144550.log.gz 2026-03-31T21:28:00.956 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144623.log 2026-03-31T21:28:00.956 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144618.log.gz 2026-03-31T21:28:00.957 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144715.log 2026-03-31T21:28:00.957 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144623.log.gz 2026-03-31T21:28:00.957 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144844.log 2026-03-31T21:28:00.958 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144715.log.gz 2026-03-31T21:28:00.958 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144843.log 2026-03-31T21:28:00.959 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144844.log.gz 2026-03-31T21:28:00.959 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144842.log 2026-03-31T21:28:00.959 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144843.log.gz 2026-03-31T21:28:00.960 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144845.log 2026-03-31T21:28:00.960 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144842.log.gz 2026-03-31T21:28:00.960 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145010.log 2026-03-31T21:28:00.961 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.144845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.144845.log.gz 2026-03-31T21:28:00.961 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145009.log 2026-03-31T21:28:00.961 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145010.log.gz 2026-03-31T21:28:00.962 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145014.log 2026-03-31T21:28:00.962 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145009.log.gz 2026-03-31T21:28:00.962 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145071.log 2026-03-31T21:28:00.963 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145014.log.gz 2026-03-31T21:28:00.963 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145156.log 2026-03-31T21:28:00.964 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145071.log.gz 2026-03-31T21:28:00.964 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145210.log 2026-03-31T21:28:00.964 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145156.log: 30.6% -- replaced with /var/log/ceph/ceph-client.admin.145156.log.gz 2026-03-31T21:28:00.965 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145220.log 2026-03-31T21:28:00.965 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145210.log.gz 2026-03-31T21:28:00.965 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145257.log 2026-03-31T21:28:00.966 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145220.log.gz 2026-03-31T21:28:00.966 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145352.log 2026-03-31T21:28:00.966 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145257.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145257.log.gz 2026-03-31T21:28:00.967 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145480.log 2026-03-31T21:28:00.967 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145352.log.gz 2026-03-31T21:28:00.967 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145482.log 2026-03-31T21:28:00.968 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145480.log.gz 2026-03-31T21:28:00.968 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145479.log 2026-03-31T21:28:00.968 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145482.log.gz 2026-03-31T21:28:00.969 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145481.log 2026-03-31T21:28:00.969 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145479.log.gz 2026-03-31T21:28:00.969 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145595.log 2026-03-31T21:28:00.970 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145481.log.gz 2026-03-31T21:28:00.970 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145667.log 2026-03-31T21:28:00.970 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145595.log: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.145595.log.gz 2026-03-31T21:28:00.971 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145666.log 2026-03-31T21:28:00.971 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145667.log.gz 2026-03-31T21:28:00.971 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145736.log 2026-03-31T21:28:00.972 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145666.log.gz 2026-03-31T21:28:00.972 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145793.log 2026-03-31T21:28:00.972 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145736.log.gz 2026-03-31T21:28:00.973 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145821.log 2026-03-31T21:28:00.973 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145793.log.gz 2026-03-31T21:28:00.973 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145887.log 2026-03-31T21:28:00.974 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145821.log.gz 2026-03-31T21:28:00.976 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145926.log 2026-03-31T21:28:00.976 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145887.log.gz 2026-03-31T21:28:00.976 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.145989.log 2026-03-31T21:28:00.977 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145926.log.gz 2026-03-31T21:28:00.977 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146038.log 2026-03-31T21:28:00.977 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.145989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.145989.log.gz 2026-03-31T21:28:00.978 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146162.log 2026-03-31T21:28:00.978 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146038.log.gz 2026-03-31T21:28:00.978 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146167.log 2026-03-31T21:28:00.979 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146162.log.gz 2026-03-31T21:28:00.979 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146163.log 2026-03-31T21:28:00.979 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146167.log.gz 2026-03-31T21:28:00.980 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146168.log 2026-03-31T21:28:00.980 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146163.log.gz 2026-03-31T21:28:00.980 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146350.log 2026-03-31T21:28:00.981 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146168.log.gz 2026-03-31T21:28:00.981 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146361.log 2026-03-31T21:28:00.981 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146350.log.gz 2026-03-31T21:28:00.982 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146360.log 2026-03-31T21:28:00.982 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146361.log.gz 2026-03-31T21:28:00.982 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146364.log 2026-03-31T21:28:00.983 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146360.log.gz 2026-03-31T21:28:00.983 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146506.log 2026-03-31T21:28:00.983 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146364.log.gz 2026-03-31T21:28:00.984 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146505.log 2026-03-31T21:28:00.984 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146506.log.gz 2026-03-31T21:28:00.984 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146579.log 2026-03-31T21:28:00.985 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146505.log.gz 2026-03-31T21:28:00.985 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146580.log 2026-03-31T21:28:00.985 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146579.log.gz 2026-03-31T21:28:00.986 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146675.log 2026-03-31T21:28:00.986 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146580.log.gz 2026-03-31T21:28:00.986 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146801.log 2026-03-31T21:28:00.987 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146675.log.gz 2026-03-31T21:28:00.987 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146802.log 2026-03-31T21:28:00.987 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146801.log.gz 2026-03-31T21:28:00.988 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146804.log 2026-03-31T21:28:00.988 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146802.log.gz 2026-03-31T21:28:00.988 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146805.log 2026-03-31T21:28:00.989 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146804.log.gz 2026-03-31T21:28:00.989 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146974.log 2026-03-31T21:28:00.989 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146805.log.gz 2026-03-31T21:28:00.990 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146973.log 2026-03-31T21:28:00.990 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146974.log.gz 2026-03-31T21:28:00.990 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146971.log 2026-03-31T21:28:00.991 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146973.log.gz 2026-03-31T21:28:00.991 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147038.log 2026-03-31T21:28:00.991 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.146971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.146971.log.gz 2026-03-31T21:28:00.991 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147116.log 2026-03-31T21:28:00.992 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147038.log.gz 2026-03-31T21:28:00.992 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147170.log 2026-03-31T21:28:00.992 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147116.log.gz 2026-03-31T21:28:00.993 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147172.log 2026-03-31T21:28:00.993 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147170.log.gz 2026-03-31T21:28:00.993 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147234.log 2026-03-31T21:28:00.994 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147172.log.gz 2026-03-31T21:28:00.994 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147312.log 2026-03-31T21:28:00.994 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147234.log.gz 2026-03-31T21:28:00.995 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147440.log 2026-03-31T21:28:00.995 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147312.log.gz 2026-03-31T21:28:00.995 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147442.log 2026-03-31T21:28:00.996 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147440.log.gz 2026-03-31T21:28:00.996 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147437.log 2026-03-31T21:28:00.996 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147442.log.gz 2026-03-31T21:28:00.997 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147439.log 2026-03-31T21:28:00.997 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147437.log.gz 2026-03-31T21:28:00.998 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147580.log 2026-03-31T21:28:00.998 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147439.log.gz 2026-03-31T21:28:00.998 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147584.log 2026-03-31T21:28:00.999 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147580.log.gz 2026-03-31T21:28:00.999 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147678.log 2026-03-31T21:28:00.999 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147584.log.gz 2026-03-31T21:28:01.000 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147675.log 2026-03-31T21:28:01.000 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147678.log.gz 2026-03-31T21:28:01.000 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147753.log 2026-03-31T21:28:01.001 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147675.log.gz 2026-03-31T21:28:01.001 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147781.log 2026-03-31T21:28:01.001 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147753.log.gz 2026-03-31T21:28:01.002 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147851.log 2026-03-31T21:28:01.002 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147781.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147781.log.gz 2026-03-31T21:28:01.002 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147855.log 2026-03-31T21:28:01.003 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147851.log.gz 2026-03-31T21:28:01.003 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147949.log 2026-03-31T21:28:01.003 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147855.log.gz 2026-03-31T21:28:01.004 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148078.log 2026-03-31T21:28:01.004 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.147949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.147949.log.gz 2026-03-31T21:28:01.004 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148077.log 2026-03-31T21:28:01.005 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148078.log.gz 2026-03-31T21:28:01.005 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148076.log 2026-03-31T21:28:01.005 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148077.log.gz 2026-03-31T21:28:01.006 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148079.log 2026-03-31T21:28:01.006 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148076.log.gz 2026-03-31T21:28:01.006 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148240.log 2026-03-31T21:28:01.007 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148079.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148079.log.gz 2026-03-31T21:28:01.007 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148248.log 2026-03-31T21:28:01.007 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148240.log.gz 2026-03-31T21:28:01.008 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148247.log 2026-03-31T21:28:01.008 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148248.log.gz 2026-03-31T21:28:01.008 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148289.log 2026-03-31T21:28:01.009 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148247.log.gz 2026-03-31T21:28:01.009 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148444.log 2026-03-31T21:28:01.009 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148289.log.gz 2026-03-31T21:28:01.010 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148442.log 2026-03-31T21:28:01.010 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148444.log.gz 2026-03-31T21:28:01.010 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148443.log 2026-03-31T21:28:01.011 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148442.log.gz 2026-03-31T21:28:01.011 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148518.log 2026-03-31T21:28:01.011 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148443.log.gz 2026-03-31T21:28:01.012 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148586.log 2026-03-31T21:28:01.012 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148518.log.gz 2026-03-31T21:28:01.012 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148715.log 2026-03-31T21:28:01.013 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148586.log.gz 2026-03-31T21:28:01.013 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148714.log 2026-03-31T21:28:01.013 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148715.log: 28.7% -- replaced with /var/log/ceph/ceph-client.admin.148715.log.gz 2026-03-31T21:28:01.014 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148716.log 2026-03-31T21:28:01.014 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148714.log.gz 2026-03-31T21:28:01.014 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148713.log 2026-03-31T21:28:01.015 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148716.log.gz 2026-03-31T21:28:01.015 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148866.log 2026-03-31T21:28:01.015 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148713.log.gz 2026-03-31T21:28:01.016 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148870.log 2026-03-31T21:28:01.016 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148866.log.gz 2026-03-31T21:28:01.016 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148873.log 2026-03-31T21:28:01.017 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148870.log.gz 2026-03-31T21:28:01.017 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148978.log 2026-03-31T21:28:01.017 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148873.log.gz 2026-03-31T21:28:01.018 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149054.log 2026-03-31T21:28:01.018 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.148978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.148978.log.gz 2026-03-31T21:28:01.018 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149053.log 2026-03-31T21:28:01.019 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149054.log.gz 2026-03-31T21:28:01.019 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149101.log 2026-03-31T21:28:01.019 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149053.log.gz 2026-03-31T21:28:01.020 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149172.log 2026-03-31T21:28:01.020 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149101.log.gz 2026-03-31T21:28:01.020 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149223.log 2026-03-31T21:28:01.021 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149172.log.gz 2026-03-31T21:28:01.021 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149351.log 2026-03-31T21:28:01.021 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149223.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.149223.log.gz 2026-03-31T21:28:01.022 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149350.log 2026-03-31T21:28:01.022 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149351.log.gz 2026-03-31T21:28:01.022 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149347.log 2026-03-31T21:28:01.023 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149350.log: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.149350.log.gz 2026-03-31T21:28:01.023 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149353.log 2026-03-31T21:28:01.023 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149347.log.gz 2026-03-31T21:28:01.024 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149456.log 2026-03-31T21:28:01.024 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149353.log.gz 2026-03-31T21:28:01.024 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149529.log 2026-03-31T21:28:01.025 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149456.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149456.log.gz 2026-03-31T21:28:01.025 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149530.log 2026-03-31T21:28:01.025 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149529.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149529.log.gz 2026-03-31T21:28:01.026 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149611.log 2026-03-31T21:28:01.026 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149530.log.gz 2026-03-31T21:28:01.026 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149664.log 2026-03-31T21:28:01.027 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149611.log.gz 2026-03-31T21:28:01.027 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149713.log 2026-03-31T21:28:01.027 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149664.log.gz 2026-03-31T21:28:01.028 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149839.log 2026-03-31T21:28:01.028 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149713.log.gz 2026-03-31T21:28:01.028 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149840.log 2026-03-31T21:28:01.029 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149839.log.gz 2026-03-31T21:28:01.029 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149842.log 2026-03-31T21:28:01.029 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149840.log.gz 2026-03-31T21:28:01.030 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149843.log 2026-03-31T21:28:01.030 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149842.log.gz 2026-03-31T21:28:01.030 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150031.log 2026-03-31T21:28:01.031 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.149843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.149843.log.gz 2026-03-31T21:28:01.031 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150034.log 2026-03-31T21:28:01.031 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150031.log.gz 2026-03-31T21:28:01.032 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150035.log 2026-03-31T21:28:01.032 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150034.log.gz 2026-03-31T21:28:01.032 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150039.log 2026-03-31T21:28:01.033 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150035.log.gz 2026-03-31T21:28:01.033 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150181.log 2026-03-31T21:28:01.033 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150039.log.gz 2026-03-31T21:28:01.034 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150180.log 2026-03-31T21:28:01.034 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150181.log.gz 2026-03-31T21:28:01.034 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150235.log 2026-03-31T21:28:01.035 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150180.log.gz 2026-03-31T21:28:01.035 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150237.log 2026-03-31T21:28:01.035 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150235.log.gz 2026-03-31T21:28:01.036 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150350.log 2026-03-31T21:28:01.036 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150237.log.gz 2026-03-31T21:28:01.036 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150379.log 2026-03-31T21:28:01.037 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150350.log.gz 2026-03-31T21:28:01.037 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150457.log 2026-03-31T21:28:01.037 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150379.log.gz 2026-03-31T21:28:01.038 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150469.log 2026-03-31T21:28:01.038 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150457.log.gz 2026-03-31T21:28:01.038 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150546.log 2026-03-31T21:28:01.039 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150469.log.gz 2026-03-31T21:28:01.039 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150675.log 2026-03-31T21:28:01.039 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150546.log.gz 2026-03-31T21:28:01.040 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150674.log 2026-03-31T21:28:01.040 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150675.log.gz 2026-03-31T21:28:01.040 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150676.log 2026-03-31T21:28:01.041 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150674.log.gz 2026-03-31T21:28:01.041 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150673.log 2026-03-31T21:28:01.041 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150676.log.gz 2026-03-31T21:28:01.042 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150789.log 2026-03-31T21:28:01.042 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150673.log.gz 2026-03-31T21:28:01.042 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150863.log 2026-03-31T21:28:01.043 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150789.log.gz 2026-03-31T21:28:01.043 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150860.log 2026-03-31T21:28:01.043 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150863.log.gz 2026-03-31T21:28:01.044 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150934.log 2026-03-31T21:28:01.044 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150860.log.gz 2026-03-31T21:28:01.044 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150987.log 2026-03-31T21:28:01.045 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150934.log.gz 2026-03-31T21:28:01.045 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151024.log 2026-03-31T21:28:01.045 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.150987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.150987.log.gz 2026-03-31T21:28:01.046 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151061.log 2026-03-31T21:28:01.046 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151024.log.gz 2026-03-31T21:28:01.046 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151130.log 2026-03-31T21:28:01.047 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151061.log.gz 2026-03-31T21:28:01.047 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151183.log 2026-03-31T21:28:01.047 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151130.log.gz 2026-03-31T21:28:01.048 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151232.log 2026-03-31T21:28:01.048 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151183.log.gz 2026-03-31T21:28:01.048 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151362.log 2026-03-31T21:28:01.049 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151232.log.gz 2026-03-31T21:28:01.049 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151359.log 2026-03-31T21:28:01.049 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151362.log.gz 2026-03-31T21:28:01.050 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151361.log 2026-03-31T21:28:01.050 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151359.log.gz 2026-03-31T21:28:01.050 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151360.log 2026-03-31T21:28:01.051 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151361.log.gz 2026-03-31T21:28:01.051 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151555.log 2026-03-31T21:28:01.051 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151360.log.gz 2026-03-31T21:28:01.052 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151557.log 2026-03-31T21:28:01.052 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151555.log.gz 2026-03-31T21:28:01.052 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151558.log 2026-03-31T21:28:01.053 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151557.log.gz 2026-03-31T21:28:01.053 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151556.log 2026-03-31T21:28:01.053 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151558.log.gz 2026-03-31T21:28:01.053 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151673.log 2026-03-31T21:28:01.054 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151556.log.gz 2026-03-31T21:28:01.054 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151752.log 2026-03-31T21:28:01.054 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151673.log.gz 2026-03-31T21:28:01.055 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151753.log 2026-03-31T21:28:01.055 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151752.log.gz 2026-03-31T21:28:01.055 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151761.log 2026-03-31T21:28:01.056 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151753.log.gz 2026-03-31T21:28:01.056 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151869.log 2026-03-31T21:28:01.056 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151761.log.gz 2026-03-31T21:28:01.057 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151897.log 2026-03-31T21:28:01.057 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151869.log.gz 2026-03-31T21:28:01.057 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151969.log 2026-03-31T21:28:01.058 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151897.log.gz 2026-03-31T21:28:01.058 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151977.log 2026-03-31T21:28:01.058 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151969.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.151969.log.gz 2026-03-31T21:28:01.059 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152065.log 2026-03-31T21:28:01.059 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.151977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.151977.log.gz 2026-03-31T21:28:01.059 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152195.log 2026-03-31T21:28:01.060 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152065.log.gz 2026-03-31T21:28:01.060 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152193.log 2026-03-31T21:28:01.060 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152195.log.gz 2026-03-31T21:28:01.061 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152194.log 2026-03-31T21:28:01.061 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152193.log.gz 2026-03-31T21:28:01.061 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152192.log 2026-03-31T21:28:01.061 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152194.log.gz 2026-03-31T21:28:01.062 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152362.log 2026-03-31T21:28:01.062 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152192.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152192.log.gz 2026-03-31T21:28:01.062 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152363.log 2026-03-31T21:28:01.063 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152362.log.gz 2026-03-31T21:28:01.063 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152364.log 2026-03-31T21:28:01.063 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152363.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152363.log.gz 2026-03-31T21:28:01.064 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152391.log 2026-03-31T21:28:01.064 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152364.log.gz 2026-03-31T21:28:01.064 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152506.log 2026-03-31T21:28:01.065 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152391.log.gz 2026-03-31T21:28:01.065 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152586.log 2026-03-31T21:28:01.065 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152506.log.gz 2026-03-31T21:28:01.066 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152587.log 2026-03-31T21:28:01.066 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152586.log.gz 2026-03-31T21:28:01.066 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152583.log 2026-03-31T21:28:01.067 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152587.log.gz 2026-03-31T21:28:01.067 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152702.log 2026-03-31T21:28:01.067 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152583.log.gz 2026-03-31T21:28:01.068 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152827.log 2026-03-31T21:28:01.068 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152702.log.gz 2026-03-31T21:28:01.068 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152826.log 2026-03-31T21:28:01.069 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152827.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152827.log.gz 2026-03-31T21:28:01.069 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152829.log 2026-03-31T21:28:01.069 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152826.log.gz 2026-03-31T21:28:01.070 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152832.log 2026-03-31T21:28:01.070 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152829.log.gz 2026-03-31T21:28:01.070 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153007.log 2026-03-31T21:28:01.070 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.152832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.152832.log.gz 2026-03-31T21:28:01.071 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153028.log 2026-03-31T21:28:01.071 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153007.log.gz 2026-03-31T21:28:01.071 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153027.log 2026-03-31T21:28:01.072 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153028.log.gz 2026-03-31T21:28:01.072 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153023.log 2026-03-31T21:28:01.072 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153027.log.gz 2026-03-31T21:28:01.073 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153143.log 2026-03-31T21:28:01.073 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153023.log.gz 2026-03-31T21:28:01.073 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153222.log 2026-03-31T21:28:01.074 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153143.log.gz 2026-03-31T21:28:01.074 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153243.log 2026-03-31T21:28:01.074 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153222.log.gz 2026-03-31T21:28:01.075 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153240.log 2026-03-31T21:28:01.075 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153243.log.gz 2026-03-31T21:28:01.075 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153339.log 2026-03-31T21:28:01.075 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153240.log.gz 2026-03-31T21:28:01.076 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153469.log 2026-03-31T21:28:01.076 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153339.log.gz 2026-03-31T21:28:01.076 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153468.log 2026-03-31T21:28:01.077 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153469.log.gz 2026-03-31T21:28:01.077 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153466.log 2026-03-31T21:28:01.077 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153468.log.gz 2026-03-31T21:28:01.078 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153467.log 2026-03-31T21:28:01.078 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153466.log.gz 2026-03-31T21:28:01.078 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153637.log 2026-03-31T21:28:01.079 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153467.log.gz 2026-03-31T21:28:01.079 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153638.log 2026-03-31T21:28:01.079 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153637.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153637.log.gz 2026-03-31T21:28:01.080 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153635.log 2026-03-31T21:28:01.080 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153638.log.gz 2026-03-31T21:28:01.080 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153703.log 2026-03-31T21:28:01.081 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153635.log.gz 2026-03-31T21:28:01.081 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153780.log 2026-03-31T21:28:01.081 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153703.log.gz 2026-03-31T21:28:01.082 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153834.log 2026-03-31T21:28:01.082 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153780.log.gz 2026-03-31T21:28:01.082 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153833.log 2026-03-31T21:28:01.083 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153834.log.gz 2026-03-31T21:28:01.083 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153901.log 2026-03-31T21:28:01.083 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153833.log.gz 2026-03-31T21:28:01.083 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153976.log 2026-03-31T21:28:01.084 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153901.log.gz 2026-03-31T21:28:01.084 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154031.log 2026-03-31T21:28:01.084 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.153976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.153976.log.gz 2026-03-31T21:28:01.085 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154032.log 2026-03-31T21:28:01.085 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154031.log.gz 2026-03-31T21:28:01.085 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154123.log 2026-03-31T21:28:01.086 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154032.log.gz 2026-03-31T21:28:01.086 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154251.log 2026-03-31T21:28:01.086 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154123.log.gz 2026-03-31T21:28:01.087 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154250.log 2026-03-31T21:28:01.087 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154251.log.gz 2026-03-31T21:28:01.087 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154253.log 2026-03-31T21:28:01.088 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154250.log.gz 2026-03-31T21:28:01.088 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154252.log 2026-03-31T21:28:01.088 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154253.log.gz 2026-03-31T21:28:01.089 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154436.log 2026-03-31T21:28:01.089 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154252.log.gz 2026-03-31T21:28:01.089 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154447.log 2026-03-31T21:28:01.090 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154436.log.gz 2026-03-31T21:28:01.090 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154446.log 2026-03-31T21:28:01.090 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154447.log.gz 2026-03-31T21:28:01.091 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154449.log 2026-03-31T21:28:01.091 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154446.log.gz 2026-03-31T21:28:01.091 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154617.log 2026-03-31T21:28:01.092 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154449.log.gz 2026-03-31T21:28:01.092 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154616.log 2026-03-31T21:28:01.092 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154617.log.gz 2026-03-31T21:28:01.093 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154618.log 2026-03-31T21:28:01.093 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154616.log.gz 2026-03-31T21:28:01.093 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154646.log 2026-03-31T21:28:01.094 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154618.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154618.log.gz 2026-03-31T21:28:01.094 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154760.log 2026-03-31T21:28:01.094 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154646.log.gz 2026-03-31T21:28:01.094 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154886.log 2026-03-31T21:28:01.095 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154760.log.gz 2026-03-31T21:28:01.095 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154889.log 2026-03-31T21:28:01.095 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154886.log.gz 2026-03-31T21:28:01.096 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154888.log 2026-03-31T21:28:01.096 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154889.log.gz 2026-03-31T21:28:01.096 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154890.log 2026-03-31T21:28:01.097 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154888.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154888.log.gz 2026-03-31T21:28:01.097 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155063.log 2026-03-31T21:28:01.097 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.154890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.154890.log.gz 2026-03-31T21:28:01.098 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155082.log 2026-03-31T21:28:01.098 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155063.log.gz 2026-03-31T21:28:01.098 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155086.log 2026-03-31T21:28:01.099 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155082.log.gz 2026-03-31T21:28:01.099 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155085.log 2026-03-31T21:28:01.099 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155086.log.gz 2026-03-31T21:28:01.100 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155201.log 2026-03-31T21:28:01.100 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155085.log.gz 2026-03-31T21:28:01.100 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155281.log 2026-03-31T21:28:01.100 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155201.log.gz 2026-03-31T21:28:01.101 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155279.log 2026-03-31T21:28:01.101 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155281.log.gz 2026-03-31T21:28:01.101 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155282.log 2026-03-31T21:28:01.102 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155279.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155279.log.gz 2026-03-31T21:28:01.102 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155397.log 2026-03-31T21:28:01.102 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155282.log.gz 2026-03-31T21:28:01.103 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155522.log 2026-03-31T21:28:01.103 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155397.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.155397.log.gz 2026-03-31T21:28:01.103 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155525.log 2026-03-31T21:28:01.104 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155522.log.gz 2026-03-31T21:28:01.104 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155524.log 2026-03-31T21:28:01.104 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155525.log.gz 2026-03-31T21:28:01.105 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155528.log 2026-03-31T21:28:01.105 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155524.log.gz 2026-03-31T21:28:01.105 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155683.log 2026-03-31T21:28:01.106 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155528.log.gz 2026-03-31T21:28:01.106 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155697.log 2026-03-31T21:28:01.106 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155683.log.gz 2026-03-31T21:28:01.106 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155694.log 2026-03-31T21:28:01.107 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155697.log.gz 2026-03-31T21:28:01.107 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155784.log 2026-03-31T21:28:01.107 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155694.log.gz 2026-03-31T21:28:01.108 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155839.log 2026-03-31T21:28:01.108 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155784.log.gz 2026-03-31T21:28:01.108 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155894.log 2026-03-31T21:28:01.109 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155839.log.gz 2026-03-31T21:28:01.109 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155895.log 2026-03-31T21:28:01.110 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155894.log.gz 2026-03-31T21:28:01.110 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155942.log 2026-03-31T21:28:01.110 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155895.log.gz 2026-03-31T21:28:01.110 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156035.log 2026-03-31T21:28:01.111 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.155942.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.155942.log.gz 2026-03-31T21:28:01.111 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156084.log 2026-03-31T21:28:01.111 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156035.log.gz 2026-03-31T21:28:01.112 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156211.log 2026-03-31T21:28:01.112 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156084.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156084.log.gz 2026-03-31T21:28:01.112 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156213.log 2026-03-31T21:28:01.113 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156211.log.gz 2026-03-31T21:28:01.113 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156214.log 2026-03-31T21:28:01.113 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156213.log.gz 2026-03-31T21:28:01.114 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156209.log 2026-03-31T21:28:01.114 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156214.log.gz 2026-03-31T21:28:01.114 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156382.log 2026-03-31T21:28:01.115 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156209.log.gz 2026-03-31T21:28:01.115 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156381.log 2026-03-31T21:28:01.115 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156382.log.gz 2026-03-31T21:28:01.116 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156383.log 2026-03-31T21:28:01.116 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156381.log.gz 2026-03-31T21:28:01.116 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156461.log 2026-03-31T21:28:01.117 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156383.log.gz 2026-03-31T21:28:01.117 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156525.log 2026-03-31T21:28:01.117 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156461.log.gz 2026-03-31T21:28:01.117 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156581.log 2026-03-31T21:28:01.118 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156525.log.gz 2026-03-31T21:28:01.118 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156579.log 2026-03-31T21:28:01.118 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156581.log.gz 2026-03-31T21:28:01.119 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156643.log 2026-03-31T21:28:01.119 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156579.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156579.log.gz 2026-03-31T21:28:01.119 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156721.log 2026-03-31T21:28:01.120 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156643.log.gz 2026-03-31T21:28:01.120 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156848.log 2026-03-31T21:28:01.120 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156721.log.gz 2026-03-31T21:28:01.121 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156851.log 2026-03-31T21:28:01.121 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156848.log.gz 2026-03-31T21:28:01.121 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156850.log 2026-03-31T21:28:01.122 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156851.log.gz 2026-03-31T21:28:01.122 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156849.log 2026-03-31T21:28:01.122 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156850.log.gz 2026-03-31T21:28:01.123 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157035.log 2026-03-31T21:28:01.123 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.156849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.156849.log.gz 2026-03-31T21:28:01.123 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157047.log 2026-03-31T21:28:01.124 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157035.log.gz 2026-03-31T21:28:01.124 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157045.log 2026-03-31T21:28:01.124 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157047.log.gz 2026-03-31T21:28:01.124 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157046.log 2026-03-31T21:28:01.125 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157045.log.gz 2026-03-31T21:28:01.125 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157162.log 2026-03-31T21:28:01.125 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157046.log.gz 2026-03-31T21:28:01.126 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157287.log 2026-03-31T21:28:01.126 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157162.log.gz 2026-03-31T21:28:01.126 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157292.log 2026-03-31T21:28:01.127 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157287.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157287.log.gz 2026-03-31T21:28:01.127 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157288.log 2026-03-31T21:28:01.127 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157292.log.gz 2026-03-31T21:28:01.128 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157290.log 2026-03-31T21:28:01.128 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157288.log.gz 2026-03-31T21:28:01.128 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157459.log 2026-03-31T21:28:01.129 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157290.log.gz 2026-03-31T21:28:01.129 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157461.log 2026-03-31T21:28:01.129 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157459.log.gz 2026-03-31T21:28:01.129 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157460.log 2026-03-31T21:28:01.130 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157461.log.gz 2026-03-31T21:28:01.130 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157512.log 2026-03-31T21:28:01.131 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157460.log.gz 2026-03-31T21:28:01.131 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157603.log 2026-03-31T21:28:01.131 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157512.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157512.log.gz 2026-03-31T21:28:01.132 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157666.log 2026-03-31T21:28:01.132 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157603.log: 27.6% -- replaced with /var/log/ceph/ceph-client.admin.157603.log.gz 2026-03-31T21:28:01.132 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157667.log 2026-03-31T21:28:01.133 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157666.log.gz 2026-03-31T21:28:01.133 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157694.log 2026-03-31T21:28:01.133 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157667.log.gz 2026-03-31T21:28:01.133 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157799.log 2026-03-31T21:28:01.134 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157694.log.gz 2026-03-31T21:28:01.134 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157927.log 2026-03-31T21:28:01.134 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157799.log.gz 2026-03-31T21:28:01.135 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157928.log 2026-03-31T21:28:01.135 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157927.log.gz 2026-03-31T21:28:01.135 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157925.log 2026-03-31T21:28:01.136 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157928.log.gz 2026-03-31T21:28:01.136 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157929.log 2026-03-31T21:28:01.136 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157925.log.gz 2026-03-31T21:28:01.137 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158080.log 2026-03-31T21:28:01.137 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.157929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.157929.log.gz 2026-03-31T21:28:01.137 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158120.log 2026-03-31T21:28:01.138 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158080.log.gz 2026-03-31T21:28:01.138 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158122.log 2026-03-31T21:28:01.138 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158120.log.gz 2026-03-31T21:28:01.139 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158125.log 2026-03-31T21:28:01.139 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158122.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158122.log.gz 2026-03-31T21:28:01.139 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158267.log 2026-03-31T21:28:01.139 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158125.log.gz 2026-03-31T21:28:01.140 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158266.log 2026-03-31T21:28:01.140 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158267.log.gz 2026-03-31T21:28:01.140 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158322.log 2026-03-31T21:28:01.141 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158266.log.gz 2026-03-31T21:28:01.141 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158319.log 2026-03-31T21:28:01.141 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158322.log.gz 2026-03-31T21:28:01.142 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158436.log 2026-03-31T21:28:01.142 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158319.log.gz 2026-03-31T21:28:01.142 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158463.log 2026-03-31T21:28:01.143 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158436.log.gz 2026-03-31T21:28:01.143 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158534.log 2026-03-31T21:28:01.143 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158463.log.gz 2026-03-31T21:28:01.144 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158557.log 2026-03-31T21:28:01.144 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158534.log.gz 2026-03-31T21:28:01.144 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158632.log 2026-03-31T21:28:01.145 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158557.log.gz 2026-03-31T21:28:01.145 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158758.log 2026-03-31T21:28:01.145 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158632.log.gz 2026-03-31T21:28:01.145 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158759.log 2026-03-31T21:28:01.146 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158758.log.gz 2026-03-31T21:28:01.146 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158761.log 2026-03-31T21:28:01.146 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158759.log.gz 2026-03-31T21:28:01.147 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158762.log 2026-03-31T21:28:01.147 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158761.log.gz 2026-03-31T21:28:01.147 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158923.log 2026-03-31T21:28:01.148 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158762.log.gz 2026-03-31T21:28:01.148 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158930.log 2026-03-31T21:28:01.148 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158923.log.gz 2026-03-31T21:28:01.149 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158931.log 2026-03-31T21:28:01.149 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158930.log.gz 2026-03-31T21:28:01.149 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159007.log 2026-03-31T21:28:01.150 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.158931.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.158931.log.gz 2026-03-31T21:28:01.150 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159073.log 2026-03-31T21:28:01.150 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159007.log.gz 2026-03-31T21:28:01.150 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159127.log 2026-03-31T21:28:01.151 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159073.log.gz 2026-03-31T21:28:01.151 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159126.log 2026-03-31T21:28:01.151 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159127.log.gz 2026-03-31T21:28:01.152 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159174.log 2026-03-31T21:28:01.152 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159126.log.gz 2026-03-31T21:28:01.153 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159269.log 2026-03-31T21:28:01.153 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159174.log.gz 2026-03-31T21:28:01.153 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159395.log 2026-03-31T21:28:01.154 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159269.log.gz 2026-03-31T21:28:01.154 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159397.log 2026-03-31T21:28:01.154 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159395.log.gz 2026-03-31T21:28:01.154 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159399.log 2026-03-31T21:28:01.155 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159397.log.gz 2026-03-31T21:28:01.155 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159394.log 2026-03-31T21:28:01.155 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159399.log.gz 2026-03-31T21:28:01.156 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159576.log 2026-03-31T21:28:01.156 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159394.log.gz 2026-03-31T21:28:01.156 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159592.log 2026-03-31T21:28:01.157 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159576.log: 30.3% -- replaced with /var/log/ceph/ceph-client.admin.159576.log.gz 2026-03-31T21:28:01.157 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159594.log 2026-03-31T21:28:01.157 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159592.log.gz 2026-03-31T21:28:01.158 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159595.log 2026-03-31T21:28:01.158 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159594.log.gz 2026-03-31T21:28:01.158 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159736.log 2026-03-31T21:28:01.159 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159595.log.gz 2026-03-31T21:28:01.159 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159737.log 2026-03-31T21:28:01.159 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159736.log.gz 2026-03-31T21:28:01.159 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159802.log 2026-03-31T21:28:01.160 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159737.log.gz 2026-03-31T21:28:01.160 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159855.log 2026-03-31T21:28:01.160 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159802.log.gz 2026-03-31T21:28:01.161 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159906.log 2026-03-31T21:28:01.161 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159855.log.gz 2026-03-31T21:28:01.161 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160036.log 2026-03-31T21:28:01.162 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.159906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.159906.log.gz 2026-03-31T21:28:01.162 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160035.log 2026-03-31T21:28:01.162 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160036.log.gz 2026-03-31T21:28:01.163 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160033.log 2026-03-31T21:28:01.163 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160035.log.gz 2026-03-31T21:28:01.163 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160034.log 2026-03-31T21:28:01.164 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160033.log.gz 2026-03-31T21:28:01.164 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160231.log 2026-03-31T21:28:01.164 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160034.log.gz 2026-03-31T21:28:01.165 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160232.log 2026-03-31T21:28:01.165 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160231.log.gz 2026-03-31T21:28:01.165 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160230.log 2026-03-31T21:28:01.166 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160232.log.gz 2026-03-31T21:28:01.166 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160229.log 2026-03-31T21:28:01.166 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160230.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160230.log.gz 2026-03-31T21:28:01.167 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160347.log 2026-03-31T21:28:01.167 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160229.log.gz 2026-03-31T21:28:01.167 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160432.log 2026-03-31T21:28:01.168 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160347.log: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.160347.log.gz 2026-03-31T21:28:01.168 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160427.log 2026-03-31T21:28:01.168 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160432.log.gz 2026-03-31T21:28:01.168 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160438.log 2026-03-31T21:28:01.169 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160427.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160427.log.gz 2026-03-31T21:28:01.169 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160595.log 2026-03-31T21:28:01.169 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160438.log.gz 2026-03-31T21:28:01.170 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160597.log 2026-03-31T21:28:01.170 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160595.log.gz 2026-03-31T21:28:01.170 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160596.log 2026-03-31T21:28:01.171 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160597.log.gz 2026-03-31T21:28:01.171 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160690.log 2026-03-31T21:28:01.171 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160596.log.gz 2026-03-31T21:28:01.172 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160818.log 2026-03-31T21:28:01.172 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160690.log.gz 2026-03-31T21:28:01.172 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160820.log 2026-03-31T21:28:01.173 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160818.log.gz 2026-03-31T21:28:01.173 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160817.log 2026-03-31T21:28:01.173 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160820.log.gz 2026-03-31T21:28:01.174 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160819.log 2026-03-31T21:28:01.174 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160817.log.gz 2026-03-31T21:28:01.174 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160933.log 2026-03-31T21:28:01.175 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160819.log.gz 2026-03-31T21:28:01.175 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161005.log 2026-03-31T21:28:01.175 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160933.log.gz 2026-03-31T21:28:01.176 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160995.log 2026-03-31T21:28:01.176 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161005.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161005.log.gz 2026-03-31T21:28:01.176 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161078.log 2026-03-31T21:28:01.177 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.160995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.160995.log.gz 2026-03-31T21:28:01.177 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161131.log 2026-03-31T21:28:01.177 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161078.log.gz 2026-03-31T21:28:01.178 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161159.log 2026-03-31T21:28:01.178 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161131.log.gz 2026-03-31T21:28:01.178 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161215.log 2026-03-31T21:28:01.179 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161159.log.gz 2026-03-31T21:28:01.179 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161274.log 2026-03-31T21:28:01.179 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161215.log.gz 2026-03-31T21:28:01.179 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161327.log 2026-03-31T21:28:01.180 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161274.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161274.log.gz 2026-03-31T21:28:01.180 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161451.log 2026-03-31T21:28:01.180 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161327.log.gz 2026-03-31T21:28:01.181 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161453.log 2026-03-31T21:28:01.181 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161451.log.gz 2026-03-31T21:28:01.181 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161455.log 2026-03-31T21:28:01.182 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161453.log.gz 2026-03-31T21:28:01.182 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161457.log 2026-03-31T21:28:01.182 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161455.log.gz 2026-03-31T21:28:01.183 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161626.log 2026-03-31T21:28:01.183 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161457.log.gz 2026-03-31T21:28:01.183 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161624.log 2026-03-31T21:28:01.184 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161626.log.gz 2026-03-31T21:28:01.184 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161625.log 2026-03-31T21:28:01.184 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161624.log.gz 2026-03-31T21:28:01.184 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161701.log 2026-03-31T21:28:01.185 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161625.log.gz 2026-03-31T21:28:01.185 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161794.log 2026-03-31T21:28:01.185 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161701.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.161701.log.gz 2026-03-31T21:28:01.186 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161795.log 2026-03-31T21:28:01.186 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161794.log.gz 2026-03-31T21:28:01.186 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161822.log 2026-03-31T21:28:01.187 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161795.log.gz 2026-03-31T21:28:01.187 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161915.log 2026-03-31T21:28:01.187 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161822.log: 28.9% -- replaced with /var/log/ceph/ceph-client.admin.161822.log.gz 2026-03-31T21:28:01.188 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162041.log 2026-03-31T21:28:01.188 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.161915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.161915.log.gz 2026-03-31T21:28:01.188 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162044.log 2026-03-31T21:28:01.189 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162041.log.gz 2026-03-31T21:28:01.189 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162042.log 2026-03-31T21:28:01.189 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162044.log.gz 2026-03-31T21:28:01.190 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162045.log 2026-03-31T21:28:01.190 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162042.log.gz 2026-03-31T21:28:01.190 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162203.log 2026-03-31T21:28:01.191 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162045.log.gz 2026-03-31T21:28:01.191 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162241.log 2026-03-31T21:28:01.191 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162203.log.gz 2026-03-31T21:28:01.191 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162240.log 2026-03-31T21:28:01.192 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162241.log.gz 2026-03-31T21:28:01.192 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162239.log 2026-03-31T21:28:01.192 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162240.log.gz 2026-03-31T21:28:01.193 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162383.log 2026-03-31T21:28:01.193 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162239.log.gz 2026-03-31T21:28:01.193 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162382.log 2026-03-31T21:28:01.194 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162383.log.gz 2026-03-31T21:28:01.194 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162443.log 2026-03-31T21:28:01.194 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162382.log.gz 2026-03-31T21:28:01.195 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162451.log 2026-03-31T21:28:01.195 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162443.log.gz 2026-03-31T21:28:01.195 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162552.log 2026-03-31T21:28:01.196 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162451.log.gz 2026-03-31T21:28:01.196 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162677.log 2026-03-31T21:28:01.196 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162552.log.gz 2026-03-31T21:28:01.197 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162678.log 2026-03-31T21:28:01.197 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162677.log.gz 2026-03-31T21:28:01.197 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162679.log 2026-03-31T21:28:01.198 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162678.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162678.log.gz 2026-03-31T21:28:01.198 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162682.log 2026-03-31T21:28:01.198 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162679.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162679.log.gz 2026-03-31T21:28:01.199 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162848.log 2026-03-31T21:28:01.199 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162682.log.gz 2026-03-31T21:28:01.199 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162847.log 2026-03-31T21:28:01.200 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162848.log.gz 2026-03-31T21:28:01.200 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162851.log 2026-03-31T21:28:01.200 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162847.log.gz 2026-03-31T21:28:01.201 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162918.log 2026-03-31T21:28:01.201 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162851.log.gz 2026-03-31T21:28:01.201 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.162993.log 2026-03-31T21:28:01.202 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162918.log.gz 2026-03-31T21:28:01.202 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163047.log 2026-03-31T21:28:01.202 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.162993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.162993.log.gz 2026-03-31T21:28:01.203 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163046.log 2026-03-31T21:28:01.203 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163047.log.gz 2026-03-31T21:28:01.203 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163120.log 2026-03-31T21:28:01.203 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163046.log.gz 2026-03-31T21:28:01.204 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163189.log 2026-03-31T21:28:01.204 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163120.log.gz 2026-03-31T21:28:01.204 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163318.log 2026-03-31T21:28:01.205 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163189.log.gz 2026-03-31T21:28:01.205 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163317.log 2026-03-31T21:28:01.205 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163318.log.gz 2026-03-31T21:28:01.206 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163315.log 2026-03-31T21:28:01.206 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163317.log.gz 2026-03-31T21:28:01.206 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163319.log 2026-03-31T21:28:01.207 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163315.log.gz 2026-03-31T21:28:01.207 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163511.log 2026-03-31T21:28:01.207 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163319.log.gz 2026-03-31T21:28:01.207 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163503.log 2026-03-31T21:28:01.208 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163511.log.gz 2026-03-31T21:28:01.208 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163514.log 2026-03-31T21:28:01.208 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163503.log.gz 2026-03-31T21:28:01.209 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163515.log 2026-03-31T21:28:01.209 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163514.log.gz 2026-03-31T21:28:01.209 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163656.log 2026-03-31T21:28:01.210 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163515.log.gz 2026-03-31T21:28:01.210 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163657.log 2026-03-31T21:28:01.210 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163656.log.gz 2026-03-31T21:28:01.211 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163719.log 2026-03-31T21:28:01.211 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163657.log.gz 2026-03-31T21:28:01.211 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163750.log 2026-03-31T21:28:01.212 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163719.log.gz 2026-03-31T21:28:01.212 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163826.log 2026-03-31T21:28:01.212 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163750.log.gz 2026-03-31T21:28:01.213 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163955.log 2026-03-31T21:28:01.213 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163826.log.gz 2026-03-31T21:28:01.213 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163956.log 2026-03-31T21:28:01.213 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163955.log.gz 2026-03-31T21:28:01.214 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163953.log 2026-03-31T21:28:01.214 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163956.log.gz 2026-03-31T21:28:01.214 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.163954.log 2026-03-31T21:28:01.215 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163953.log.gz 2026-03-31T21:28:01.215 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164071.log 2026-03-31T21:28:01.215 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.163954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.163954.log.gz 2026-03-31T21:28:01.216 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164160.log 2026-03-31T21:28:01.216 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164071.log.gz 2026-03-31T21:28:01.216 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164161.log 2026-03-31T21:28:01.217 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164160.log.gz 2026-03-31T21:28:01.217 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164162.log 2026-03-31T21:28:01.217 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164161.log.gz 2026-03-31T21:28:01.218 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164267.log 2026-03-31T21:28:01.218 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164162.log.gz 2026-03-31T21:28:01.218 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164322.log 2026-03-31T21:28:01.219 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164267.log.gz 2026-03-31T21:28:01.219 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164325.log 2026-03-31T21:28:01.219 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164322.log.gz 2026-03-31T21:28:01.220 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164369.log 2026-03-31T21:28:01.220 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164325.log.gz 2026-03-31T21:28:01.220 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164463.log 2026-03-31T21:28:01.221 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164369.log.gz 2026-03-31T21:28:01.221 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164589.log 2026-03-31T21:28:01.221 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164463.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164463.log.gz 2026-03-31T21:28:01.221 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164593.log 2026-03-31T21:28:01.222 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164589.log.gz 2026-03-31T21:28:01.222 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164592.log 2026-03-31T21:28:01.222 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164593.log.gz 2026-03-31T21:28:01.223 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164590.log 2026-03-31T21:28:01.223 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164592.log.gz 2026-03-31T21:28:01.223 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164782.log 2026-03-31T21:28:01.224 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164590.log.gz 2026-03-31T21:28:01.224 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164778.log 2026-03-31T21:28:01.224 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164782.log.gz 2026-03-31T21:28:01.225 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164787.log 2026-03-31T21:28:01.225 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164778.log.gz 2026-03-31T21:28:01.225 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164789.log 2026-03-31T21:28:01.226 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164787.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164787.log.gz 2026-03-31T21:28:01.226 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164904.log 2026-03-31T21:28:01.226 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164789.log.gz 2026-03-31T21:28:01.227 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164985.log 2026-03-31T21:28:01.227 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164904.log.gz 2026-03-31T21:28:01.227 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164984.log 2026-03-31T21:28:01.227 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164985.log.gz 2026-03-31T21:28:01.228 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.164983.log 2026-03-31T21:28:01.228 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164984.log.gz 2026-03-31T21:28:01.228 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165100.log 2026-03-31T21:28:01.229 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.164983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.164983.log.gz 2026-03-31T21:28:01.229 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165127.log 2026-03-31T21:28:01.229 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165100.log.gz 2026-03-31T21:28:01.230 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165198.log 2026-03-31T21:28:01.230 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165127.log.gz 2026-03-31T21:28:01.230 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165211.log 2026-03-31T21:28:01.231 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165198.log.gz 2026-03-31T21:28:01.231 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165296.log 2026-03-31T21:28:01.231 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165211.log.gz 2026-03-31T21:28:01.232 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165420.log 2026-03-31T21:28:01.232 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165296.log.gz 2026-03-31T21:28:01.232 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165423.log 2026-03-31T21:28:01.233 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165420.log.gz 2026-03-31T21:28:01.233 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165421.log 2026-03-31T21:28:01.233 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165423.log.gz 2026-03-31T21:28:01.234 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165426.log 2026-03-31T21:28:01.234 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165421.log.gz 2026-03-31T21:28:01.234 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165539.log 2026-03-31T21:28:01.235 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165426.log.gz 2026-03-31T21:28:01.235 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165611.log 2026-03-31T21:28:01.235 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165539.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165539.log.gz 2026-03-31T21:28:01.236 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165609.log 2026-03-31T21:28:01.236 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165611.log.gz 2026-03-31T21:28:01.236 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165684.log 2026-03-31T21:28:01.237 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165609.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165609.log.gz 2026-03-31T21:28:01.237 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165737.log 2026-03-31T21:28:01.237 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165684.log.gz 2026-03-31T21:28:01.238 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165784.log 2026-03-31T21:28:01.238 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165737.log.gz 2026-03-31T21:28:01.238 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165811.log 2026-03-31T21:28:01.239 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165784.log.gz 2026-03-31T21:28:01.239 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165880.log 2026-03-31T21:28:01.239 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165811.log.gz 2026-03-31T21:28:01.240 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.165933.log 2026-03-31T21:28:01.240 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165880.log.gz 2026-03-31T21:28:01.240 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166062.log 2026-03-31T21:28:01.241 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.165933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.165933.log.gz 2026-03-31T21:28:01.241 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166060.log 2026-03-31T21:28:01.241 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166062.log.gz 2026-03-31T21:28:01.242 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166059.log 2026-03-31T21:28:01.242 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166060.log.gz 2026-03-31T21:28:01.242 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166063.log 2026-03-31T21:28:01.243 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166059.log.gz 2026-03-31T21:28:01.243 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166246.log 2026-03-31T21:28:01.243 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166063.log.gz 2026-03-31T21:28:01.244 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166254.log 2026-03-31T21:28:01.244 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166246.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166246.log.gz 2026-03-31T21:28:01.244 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166258.log 2026-03-31T21:28:01.245 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166254.log.gz 2026-03-31T21:28:01.245 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166259.log 2026-03-31T21:28:01.245 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166258.log.gz 2026-03-31T21:28:01.246 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166400.log 2026-03-31T21:28:01.246 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166259.log.gz 2026-03-31T21:28:01.246 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166401.log 2026-03-31T21:28:01.247 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166400.log: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.166400.log.gz 2026-03-31T21:28:01.247 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166474.log 2026-03-31T21:28:01.247 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166401.log.gz 2026-03-31T21:28:01.248 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166480.log 2026-03-31T21:28:01.248 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166474.log.gz 2026-03-31T21:28:01.248 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166570.log 2026-03-31T21:28:01.249 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166480.log.gz 2026-03-31T21:28:01.249 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166698.log 2026-03-31T21:28:01.249 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166570.log.gz 2026-03-31T21:28:01.250 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166700.log 2026-03-31T21:28:01.250 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166698.log.gz 2026-03-31T21:28:01.250 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166699.log 2026-03-31T21:28:01.251 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166700.log.gz 2026-03-31T21:28:01.251 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166696.log 2026-03-31T21:28:01.251 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166699.log.gz 2026-03-31T21:28:01.252 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166884.log 2026-03-31T21:28:01.252 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166696.log.gz 2026-03-31T21:28:01.252 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166892.log 2026-03-31T21:28:01.253 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166884.log.gz 2026-03-31T21:28:01.253 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166893.log 2026-03-31T21:28:01.253 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166892.log.gz 2026-03-31T21:28:01.254 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.166896.log 2026-03-31T21:28:01.254 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166893.log.gz 2026-03-31T21:28:01.254 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167064.log 2026-03-31T21:28:01.255 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.166896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.166896.log.gz 2026-03-31T21:28:01.255 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167063.log 2026-03-31T21:28:01.255 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167064.log.gz 2026-03-31T21:28:01.256 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167065.log 2026-03-31T21:28:01.256 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167063.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167063.log.gz 2026-03-31T21:28:01.256 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167138.log 2026-03-31T21:28:01.257 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167065.log.gz 2026-03-31T21:28:01.257 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167207.log 2026-03-31T21:28:01.257 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167138.log.gz 2026-03-31T21:28:01.258 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167335.log 2026-03-31T21:28:01.258 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167207.log.gz 2026-03-31T21:28:01.258 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167336.log 2026-03-31T21:28:01.259 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167335.log: 30.9% -- replaced with /var/log/ceph/ceph-client.admin.167335.log.gz 2026-03-31T21:28:01.259 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167337.log 2026-03-31T21:28:01.259 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167336.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167336.log.gz 2026-03-31T21:28:01.260 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167334.log 2026-03-31T21:28:01.260 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167337.log.gz 2026-03-31T21:28:01.260 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167440.log 2026-03-31T21:28:01.261 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167334.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167334.log.gz 2026-03-31T21:28:01.261 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167513.log 2026-03-31T21:28:01.261 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167440.log.gz 2026-03-31T21:28:01.262 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167501.log 2026-03-31T21:28:01.262 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167513.log.gz 2026-03-31T21:28:01.262 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167599.log 2026-03-31T21:28:01.263 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167501.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167501.log.gz 2026-03-31T21:28:01.263 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167648.log 2026-03-31T21:28:01.263 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167599.log.gz 2026-03-31T21:28:01.264 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167703.log 2026-03-31T21:28:01.264 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167648.log.gz 2026-03-31T21:28:01.264 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167704.log 2026-03-31T21:28:01.265 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167703.log.gz 2026-03-31T21:28:01.265 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167795.log 2026-03-31T21:28:01.265 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167704.log.gz 2026-03-31T21:28:01.266 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167924.log 2026-03-31T21:28:01.266 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167795.log.gz 2026-03-31T21:28:01.266 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167923.log 2026-03-31T21:28:01.267 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167924.log.gz 2026-03-31T21:28:01.267 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167920.log 2026-03-31T21:28:01.267 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167923.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167923.log.gz 2026-03-31T21:28:01.268 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.167925.log 2026-03-31T21:28:01.268 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167920.log.gz 2026-03-31T21:28:01.268 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168118.log 2026-03-31T21:28:01.269 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.167925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.167925.log.gz 2026-03-31T21:28:01.269 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168117.log 2026-03-31T21:28:01.269 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168118.log.gz 2026-03-31T21:28:01.270 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168121.log 2026-03-31T21:28:01.270 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168117.log.gz 2026-03-31T21:28:01.270 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168116.log 2026-03-31T21:28:01.271 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168121.log.gz 2026-03-31T21:28:01.271 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168236.log 2026-03-31T21:28:01.271 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168116.log.gz 2026-03-31T21:28:01.272 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168319.log 2026-03-31T21:28:01.272 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168236.log.gz 2026-03-31T21:28:01.272 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168317.log 2026-03-31T21:28:01.273 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168319.log.gz 2026-03-31T21:28:01.273 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168318.log 2026-03-31T21:28:01.273 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168317.log.gz 2026-03-31T21:28:01.274 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168432.log 2026-03-31T21:28:01.274 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168318.log.gz 2026-03-31T21:28:01.274 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168499.log 2026-03-31T21:28:01.275 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168432.log.gz 2026-03-31T21:28:01.275 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168484.log 2026-03-31T21:28:01.275 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168499.log.gz 2026-03-31T21:28:01.276 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168533.log 2026-03-31T21:28:01.276 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168484.log.gz 2026-03-31T21:28:01.276 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168628.log 2026-03-31T21:28:01.277 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168533.log.gz 2026-03-31T21:28:01.277 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168754.log 2026-03-31T21:28:01.277 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168628.log.gz 2026-03-31T21:28:01.278 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168755.log 2026-03-31T21:28:01.278 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168754.log.gz 2026-03-31T21:28:01.278 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168752.log 2026-03-31T21:28:01.279 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168755.log.gz 2026-03-31T21:28:01.279 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168758.log 2026-03-31T21:28:01.279 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168752.log.gz 2026-03-31T21:28:01.279 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168871.log 2026-03-31T21:28:01.280 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168758.log.gz 2026-03-31T21:28:01.280 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168944.log 2026-03-31T21:28:01.281 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168871.log.gz 2026-03-31T21:28:01.281 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.168945.log 2026-03-31T21:28:01.281 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168944.log.gz 2026-03-31T21:28:01.281 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169016.log 2026-03-31T21:28:01.282 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.168945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.168945.log.gz 2026-03-31T21:28:01.282 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169069.log 2026-03-31T21:28:01.283 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169016.log.gz 2026-03-31T21:28:01.283 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169098.log 2026-03-31T21:28:01.283 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169069.log.gz 2026-03-31T21:28:01.284 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169150.log 2026-03-31T21:28:01.284 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169098.log.gz 2026-03-31T21:28:01.284 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169212.log 2026-03-31T21:28:01.285 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169150.log.gz 2026-03-31T21:28:01.285 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169265.log 2026-03-31T21:28:01.285 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169212.log.gz 2026-03-31T21:28:01.286 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169312.log 2026-03-31T21:28:01.286 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169265.log.gz 2026-03-31T21:28:01.286 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169339.log 2026-03-31T21:28:01.287 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169312.log.gz 2026-03-31T21:28:01.287 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169398.log 2026-03-31T21:28:01.287 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169339.log: 29.6% -- replaced with /var/log/ceph/ceph-client.admin.169339.log.gz 2026-03-31T21:28:01.288 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169461.log 2026-03-31T21:28:01.288 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169398.log.gz 2026-03-31T21:28:01.288 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169589.log 2026-03-31T21:28:01.289 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169461.log.gz 2026-03-31T21:28:01.289 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169590.log 2026-03-31T21:28:01.289 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169589.log.gz 2026-03-31T21:28:01.290 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169591.log 2026-03-31T21:28:01.290 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169590.log.gz 2026-03-31T21:28:01.290 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169588.log 2026-03-31T21:28:01.291 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169591.log.gz 2026-03-31T21:28:01.291 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169758.log 2026-03-31T21:28:01.291 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169588.log.gz 2026-03-31T21:28:01.292 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169760.log 2026-03-31T21:28:01.292 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169758.log.gz 2026-03-31T21:28:01.292 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169757.log 2026-03-31T21:28:01.293 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169760.log.gz 2026-03-31T21:28:01.293 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169845.log 2026-03-31T21:28:01.293 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169757.log.gz 2026-03-31T21:28:01.294 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169928.log 2026-03-31T21:28:01.294 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169845.log.gz 2026-03-31T21:28:01.294 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169929.log 2026-03-31T21:28:01.295 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169928.log.gz 2026-03-31T21:28:01.295 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.169968.log 2026-03-31T21:28:01.295 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169929.log.gz 2026-03-31T21:28:01.296 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170033.log 2026-03-31T21:28:01.296 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.169968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.169968.log.gz 2026-03-31T21:28:01.296 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170098.log 2026-03-31T21:28:01.297 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170033.log.gz 2026-03-31T21:28:01.297 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170127.log 2026-03-31T21:28:01.297 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170098.log.gz 2026-03-31T21:28:01.298 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170196.log 2026-03-31T21:28:01.298 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170127.log.gz 2026-03-31T21:28:01.298 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170324.log 2026-03-31T21:28:01.299 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170196.log.gz 2026-03-31T21:28:01.299 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170325.log 2026-03-31T21:28:01.299 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170324.log.gz 2026-03-31T21:28:01.300 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170323.log 2026-03-31T21:28:01.300 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170325.log.gz 2026-03-31T21:28:01.300 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170326.log 2026-03-31T21:28:01.301 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170323.log.gz 2026-03-31T21:28:01.301 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170504.log 2026-03-31T21:28:01.301 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170326.log.gz 2026-03-31T21:28:01.302 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170522.log 2026-03-31T21:28:01.302 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170504.log.gz 2026-03-31T21:28:01.302 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170521.log 2026-03-31T21:28:01.303 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170522.log.gz 2026-03-31T21:28:01.303 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170520.log 2026-03-31T21:28:01.303 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170521.log.gz 2026-03-31T21:28:01.304 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170664.log 2026-03-31T21:28:01.304 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170520.log.gz 2026-03-31T21:28:01.304 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170663.log 2026-03-31T21:28:01.305 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170664.log.gz 2026-03-31T21:28:01.305 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170737.log 2026-03-31T21:28:01.305 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170663.log.gz 2026-03-31T21:28:01.306 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170742.log 2026-03-31T21:28:01.306 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170737.log.gz 2026-03-31T21:28:01.306 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170833.log 2026-03-31T21:28:01.307 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170742.log.gz 2026-03-31T21:28:01.307 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.170882.log 2026-03-31T21:28:01.308 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170833.log.gz 2026-03-31T21:28:01.308 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171009.log 2026-03-31T21:28:01.308 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.170882.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.170882.log.gz 2026-03-31T21:28:01.308 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171010.log 2026-03-31T21:28:01.309 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171009.log.gz 2026-03-31T21:28:01.309 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171012.log 2026-03-31T21:28:01.310 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171010.log.gz 2026-03-31T21:28:01.310 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171008.log 2026-03-31T21:28:01.310 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171012.log.gz 2026-03-31T21:28:01.311 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171196.log 2026-03-31T21:28:01.311 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171008.log.gz 2026-03-31T21:28:01.311 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171208.log 2026-03-31T21:28:01.312 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171196.log.gz 2026-03-31T21:28:01.312 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171206.log 2026-03-31T21:28:01.312 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171208.log.gz 2026-03-31T21:28:01.313 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171207.log 2026-03-31T21:28:01.313 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171206.log.gz 2026-03-31T21:28:01.313 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171375.log 2026-03-31T21:28:01.314 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171207.log.gz 2026-03-31T21:28:01.314 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171377.log 2026-03-31T21:28:01.314 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171375.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171375.log.gz 2026-03-31T21:28:01.315 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171376.log 2026-03-31T21:28:01.315 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171377.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171377.log.gz 2026-03-31T21:28:01.315 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171454.log 2026-03-31T21:28:01.316 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171376.log.gz 2026-03-31T21:28:01.316 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171545.log 2026-03-31T21:28:01.316 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171454.log.gz 2026-03-31T21:28:01.317 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171546.log 2026-03-31T21:28:01.317 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171545.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171545.log.gz 2026-03-31T21:28:01.317 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171610.log 2026-03-31T21:28:01.318 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171546.log.gz 2026-03-31T21:28:01.318 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171617.log 2026-03-31T21:28:01.318 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171610.log.gz 2026-03-31T21:28:01.319 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171715.log 2026-03-31T21:28:01.319 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171617.log.gz 2026-03-31T21:28:01.319 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171842.log 2026-03-31T21:28:01.320 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171715.log.gz 2026-03-31T21:28:01.320 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171845.log 2026-03-31T21:28:01.320 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171842.log.gz 2026-03-31T21:28:01.321 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171844.log 2026-03-31T21:28:01.321 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171845.log.gz 2026-03-31T21:28:01.321 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.171843.log 2026-03-31T21:28:01.322 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171844.log.gz 2026-03-31T21:28:01.322 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172014.log 2026-03-31T21:28:01.322 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.171843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.171843.log.gz 2026-03-31T21:28:01.323 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172012.log 2026-03-31T21:28:01.323 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172014.log.gz 2026-03-31T21:28:01.323 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172013.log 2026-03-31T21:28:01.324 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172012.log.gz 2026-03-31T21:28:01.324 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172071.log 2026-03-31T21:28:01.324 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172013.log: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.172013.log.gz 2026-03-31T21:28:01.325 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172182.log 2026-03-31T21:28:01.325 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172071.log.gz 2026-03-31T21:28:01.325 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172183.log 2026-03-31T21:28:01.326 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172182.log.gz 2026-03-31T21:28:01.326 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172249.log 2026-03-31T21:28:01.326 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172183.log.gz 2026-03-31T21:28:01.327 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172301.log 2026-03-31T21:28:01.327 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172249.log.gz 2026-03-31T21:28:01.327 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172352.log 2026-03-31T21:28:01.328 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172301.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172301.log.gz 2026-03-31T21:28:01.328 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172380.log 2026-03-31T21:28:01.328 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172352.log.gz 2026-03-31T21:28:01.329 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172436.log 2026-03-31T21:28:01.329 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172380.log.gz 2026-03-31T21:28:01.329 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172499.log 2026-03-31T21:28:01.330 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172436.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172436.log.gz 2026-03-31T21:28:01.330 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172626.log 2026-03-31T21:28:01.331 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172499.log.gz 2026-03-31T21:28:01.331 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172628.log 2026-03-31T21:28:01.331 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172626.log.gz 2026-03-31T21:28:01.332 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172627.log 2026-03-31T21:28:01.332 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172628.log.gz 2026-03-31T21:28:01.332 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172629.log 2026-03-31T21:28:01.333 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172627.log.gz 2026-03-31T21:28:01.333 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172823.log 2026-03-31T21:28:01.333 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172629.log.gz 2026-03-31T21:28:01.334 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172824.log 2026-03-31T21:28:01.334 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172823.log.gz 2026-03-31T21:28:01.334 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172821.log 2026-03-31T21:28:01.335 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172824.log.gz 2026-03-31T21:28:01.335 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172825.log 2026-03-31T21:28:01.335 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172821.log.gz 2026-03-31T21:28:01.336 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.172940.log 2026-03-31T21:28:01.336 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172825.log.gz 2026-03-31T21:28:01.336 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173031.log 2026-03-31T21:28:01.337 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.172940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.172940.log.gz 2026-03-31T21:28:01.337 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173024.log 2026-03-31T21:28:01.337 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173031.log.gz 2026-03-31T21:28:01.338 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173023.log 2026-03-31T21:28:01.338 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173024.log.gz 2026-03-31T21:28:01.338 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173136.log 2026-03-31T21:28:01.339 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173023.log.gz 2026-03-31T21:28:01.339 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173264.log 2026-03-31T21:28:01.339 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173136.log.gz 2026-03-31T21:28:01.340 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173266.log 2026-03-31T21:28:01.340 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173264.log.gz 2026-03-31T21:28:01.340 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173265.log 2026-03-31T21:28:01.341 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173266.log.gz 2026-03-31T21:28:01.341 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173263.log 2026-03-31T21:28:01.341 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173265.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173265.log.gz 2026-03-31T21:28:01.342 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173381.log 2026-03-31T21:28:01.342 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173263.log.gz 2026-03-31T21:28:01.342 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173454.log 2026-03-31T21:28:01.343 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173381.log.gz 2026-03-31T21:28:01.343 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173455.log 2026-03-31T21:28:01.343 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173454.log.gz 2026-03-31T21:28:01.344 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173494.log 2026-03-31T21:28:01.344 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173455.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173455.log.gz 2026-03-31T21:28:01.344 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173577.log 2026-03-31T21:28:01.345 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173494.log.gz 2026-03-31T21:28:01.345 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173614.log 2026-03-31T21:28:01.345 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173577.log.gz 2026-03-31T21:28:01.346 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173680.log 2026-03-31T21:28:01.346 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173614.log.gz 2026-03-31T21:28:01.346 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173681.log 2026-03-31T21:28:01.347 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173680.log.gz 2026-03-31T21:28:01.347 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173773.log 2026-03-31T21:28:01.347 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173681.log.gz 2026-03-31T21:28:01.348 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173900.log 2026-03-31T21:28:01.348 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173773.log.gz 2026-03-31T21:28:01.348 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173901.log 2026-03-31T21:28:01.349 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173900.log.gz 2026-03-31T21:28:01.349 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173902.log 2026-03-31T21:28:01.349 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173901.log.gz 2026-03-31T21:28:01.350 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.173903.log 2026-03-31T21:28:01.350 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173902.log.gz 2026-03-31T21:28:01.350 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174082.log 2026-03-31T21:28:01.351 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.173903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.173903.log.gz 2026-03-31T21:28:01.351 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174096.log 2026-03-31T21:28:01.351 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174082.log.gz 2026-03-31T21:28:01.352 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174099.log 2026-03-31T21:28:01.352 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174096.log.gz 2026-03-31T21:28:01.352 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174098.log 2026-03-31T21:28:01.353 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174099.log.gz 2026-03-31T21:28:01.353 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174214.log 2026-03-31T21:28:01.354 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174098.log.gz 2026-03-31T21:28:01.354 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174305.log 2026-03-31T21:28:01.354 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174214.log.gz 2026-03-31T21:28:01.355 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174304.log 2026-03-31T21:28:01.355 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174305.log.gz 2026-03-31T21:28:01.355 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174292.log 2026-03-31T21:28:01.356 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174304.log.gz 2026-03-31T21:28:01.356 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174410.log 2026-03-31T21:28:01.356 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174292.log.gz 2026-03-31T21:28:01.356 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174466.log 2026-03-31T21:28:01.357 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174410.log.gz 2026-03-31T21:28:01.357 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174462.log 2026-03-31T21:28:01.357 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174466.log.gz 2026-03-31T21:28:01.358 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174549.log 2026-03-31T21:28:01.358 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174462.log.gz 2026-03-31T21:28:01.358 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174606.log 2026-03-31T21:28:01.359 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174549.log.gz 2026-03-31T21:28:01.359 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174732.log 2026-03-31T21:28:01.359 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174606.log.gz 2026-03-31T21:28:01.360 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174733.log 2026-03-31T21:28:01.360 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174732.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174732.log.gz 2026-03-31T21:28:01.360 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174731.log 2026-03-31T21:28:01.361 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174733.log.gz 2026-03-31T21:28:01.361 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174736.log 2026-03-31T21:28:01.361 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174731.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174731.log.gz 2026-03-31T21:28:01.362 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174849.log 2026-03-31T21:28:01.362 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174736.log.gz 2026-03-31T21:28:01.362 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174915.log 2026-03-31T21:28:01.363 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174849.log.gz 2026-03-31T21:28:01.363 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174922.log 2026-03-31T21:28:01.363 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174915.log.gz 2026-03-31T21:28:01.364 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.174994.log 2026-03-31T21:28:01.364 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174922.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174922.log.gz 2026-03-31T21:28:01.364 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175047.log 2026-03-31T21:28:01.365 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.174994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.174994.log.gz 2026-03-31T21:28:01.365 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175075.log 2026-03-31T21:28:01.365 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175047.log.gz 2026-03-31T21:28:01.366 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175123.log 2026-03-31T21:28:01.366 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175075.log.gz 2026-03-31T21:28:01.366 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175190.log 2026-03-31T21:28:01.367 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175123.log.gz 2026-03-31T21:28:01.367 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175243.log 2026-03-31T21:28:01.367 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175190.log.gz 2026-03-31T21:28:01.368 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175370.log 2026-03-31T21:28:01.368 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175243.log.gz 2026-03-31T21:28:01.368 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175371.log 2026-03-31T21:28:01.369 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175370.log.gz 2026-03-31T21:28:01.369 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175372.log 2026-03-31T21:28:01.369 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175371.log.gz 2026-03-31T21:28:01.369 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175373.log 2026-03-31T21:28:01.370 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175372.log.gz 2026-03-31T21:28:01.370 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175486.log 2026-03-31T21:28:01.371 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175373.log.gz 2026-03-31T21:28:01.371 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175558.log 2026-03-31T21:28:01.371 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175486.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175486.log.gz 2026-03-31T21:28:01.371 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175557.log 2026-03-31T21:28:01.372 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175558.log.gz 2026-03-31T21:28:01.372 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175631.log 2026-03-31T21:28:01.372 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175557.log.gz 2026-03-31T21:28:01.373 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175684.log 2026-03-31T21:28:01.373 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175631.log.gz 2026-03-31T21:28:01.373 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175756.log 2026-03-31T21:28:01.374 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175684.log.gz 2026-03-31T21:28:01.374 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175737.log 2026-03-31T21:28:01.374 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175756.log.gz 2026-03-31T21:28:01.375 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175827.log 2026-03-31T21:28:01.375 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175737.log.gz 2026-03-31T21:28:01.375 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.175880.log 2026-03-31T21:28:01.376 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175827.log: 27.6% -- replaced with /var/log/ceph/ceph-client.admin.175827.log.gz 2026-03-31T21:28:01.376 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176007.log 2026-03-31T21:28:01.376 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.175880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.175880.log.gz 2026-03-31T21:28:01.377 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176008.log 2026-03-31T21:28:01.377 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176007.log.gz 2026-03-31T21:28:01.378 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176010.log 2026-03-31T21:28:01.378 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176008.log.gz 2026-03-31T21:28:01.378 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176009.log 2026-03-31T21:28:01.379 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176010.log.gz 2026-03-31T21:28:01.379 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176189.log 2026-03-31T21:28:01.379 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176009.log.gz 2026-03-31T21:28:01.380 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176206.log 2026-03-31T21:28:01.380 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176189.log.gz 2026-03-31T21:28:01.380 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176205.log 2026-03-31T21:28:01.381 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176206.log.gz 2026-03-31T21:28:01.381 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176204.log 2026-03-31T21:28:01.381 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176205.log.gz 2026-03-31T21:28:01.381 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176399.log 2026-03-31T21:28:01.382 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176204.log.gz 2026-03-31T21:28:01.382 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176400.log 2026-03-31T21:28:01.382 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176399.log.gz 2026-03-31T21:28:01.383 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176402.log 2026-03-31T21:28:01.383 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176400.log.gz 2026-03-31T21:28:01.383 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176401.log 2026-03-31T21:28:01.384 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176402.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176402.log.gz 2026-03-31T21:28:01.384 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176517.log 2026-03-31T21:28:01.384 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176401.log.gz 2026-03-31T21:28:01.385 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176566.log 2026-03-31T21:28:01.385 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176517.log.gz 2026-03-31T21:28:01.385 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176615.log 2026-03-31T21:28:01.386 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176566.log.gz 2026-03-31T21:28:01.386 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176664.log 2026-03-31T21:28:01.386 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176615.log.gz 2026-03-31T21:28:01.387 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176713.log 2026-03-31T21:28:01.387 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176664.log.gz 2026-03-31T21:28:01.387 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176837.log 2026-03-31T21:28:01.388 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176713.log.gz 2026-03-31T21:28:01.388 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176842.log 2026-03-31T21:28:01.388 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176837.log.gz 2026-03-31T21:28:01.389 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176841.log 2026-03-31T21:28:01.389 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176842.log.gz 2026-03-31T21:28:01.389 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.176843.log 2026-03-31T21:28:01.390 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176841.log.gz 2026-03-31T21:28:01.390 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177004.log 2026-03-31T21:28:01.390 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.176843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.176843.log.gz 2026-03-31T21:28:01.391 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177010.log 2026-03-31T21:28:01.391 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177004.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177004.log.gz 2026-03-31T21:28:01.391 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177012.log 2026-03-31T21:28:01.392 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177010.log.gz 2026-03-31T21:28:01.392 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177089.log 2026-03-31T21:28:01.392 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177012.log.gz 2026-03-31T21:28:01.393 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177154.log 2026-03-31T21:28:01.393 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177089.log.gz 2026-03-31T21:28:01.393 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177208.log 2026-03-31T21:28:01.394 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177154.log.gz 2026-03-31T21:28:01.394 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177228.log 2026-03-31T21:28:01.394 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177208.log.gz 2026-03-31T21:28:01.395 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177266.log 2026-03-31T21:28:01.395 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177228.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177228.log.gz 2026-03-31T21:28:01.395 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177350.log 2026-03-31T21:28:01.396 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177266.log.gz 2026-03-31T21:28:01.396 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177397.log 2026-03-31T21:28:01.396 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177350.log.gz 2026-03-31T21:28:01.396 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177442.log 2026-03-31T21:28:01.397 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177397.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177397.log.gz 2026-03-31T21:28:01.397 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177471.log 2026-03-31T21:28:01.397 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177442.log.gz 2026-03-31T21:28:01.398 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177546.log 2026-03-31T21:28:01.398 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177471.log.gz 2026-03-31T21:28:01.399 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177595.log 2026-03-31T21:28:01.399 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177546.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177546.log.gz 2026-03-31T21:28:01.399 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177644.log 2026-03-31T21:28:01.400 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177595.log.gz 2026-03-31T21:28:01.400 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177693.log 2026-03-31T21:28:01.400 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177644.log.gz 2026-03-31T21:28:01.401 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177742.log 2026-03-31T21:28:01.401 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177693.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177693.log.gz 2026-03-31T21:28:01.401 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177791.log 2026-03-31T21:28:01.402 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177742.log.gz 2026-03-31T21:28:01.402 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177840.log 2026-03-31T21:28:01.402 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177791.log.gz 2026-03-31T21:28:01.403 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177889.log 2026-03-31T21:28:01.403 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177840.log.gz 2026-03-31T21:28:01.403 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177938.log 2026-03-31T21:28:01.404 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177889.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177889.log.gz 2026-03-31T21:28:01.404 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.177987.log 2026-03-31T21:28:01.404 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177938.log.gz 2026-03-31T21:28:01.404 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178036.log 2026-03-31T21:28:01.405 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.177987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.177987.log.gz 2026-03-31T21:28:01.405 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178085.log 2026-03-31T21:28:01.405 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178036.log.gz 2026-03-31T21:28:01.406 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178134.log 2026-03-31T21:28:01.406 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178085.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178085.log.gz 2026-03-31T21:28:01.406 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178183.log 2026-03-31T21:28:01.407 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178134.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178134.log.gz 2026-03-31T21:28:01.407 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178232.log 2026-03-31T21:28:01.407 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178183.log.gz 2026-03-31T21:28:01.408 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178283.log 2026-03-31T21:28:01.408 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178232.log.gz 2026-03-31T21:28:01.408 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178332.log 2026-03-31T21:28:01.409 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178283.log.gz 2026-03-31T21:28:01.409 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178381.log 2026-03-31T21:28:01.409 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178332.log.gz 2026-03-31T21:28:01.410 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178430.log 2026-03-31T21:28:01.410 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178381.log.gz 2026-03-31T21:28:01.410 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178479.log 2026-03-31T21:28:01.411 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178430.log.gz 2026-03-31T21:28:01.411 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178528.log 2026-03-31T21:28:01.411 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178479.log.gz 2026-03-31T21:28:01.412 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178577.log 2026-03-31T21:28:01.412 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178528.log.gz 2026-03-31T21:28:01.412 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178626.log 2026-03-31T21:28:01.413 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178577.log.gz 2026-03-31T21:28:01.413 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178675.log 2026-03-31T21:28:01.413 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178626.log.gz 2026-03-31T21:28:01.414 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178724.log 2026-03-31T21:28:01.414 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178675.log.gz 2026-03-31T21:28:01.414 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178773.log 2026-03-31T21:28:01.415 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178724.log.gz 2026-03-31T21:28:01.415 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178822.log 2026-03-31T21:28:01.415 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178773.log.gz 2026-03-31T21:28:01.416 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178871.log 2026-03-31T21:28:01.416 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178822.log.gz 2026-03-31T21:28:01.416 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178920.log 2026-03-31T21:28:01.417 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178871.log.gz 2026-03-31T21:28:01.417 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.178969.log 2026-03-31T21:28:01.417 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178920.log.gz 2026-03-31T21:28:01.417 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179018.log 2026-03-31T21:28:01.418 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.178969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.178969.log.gz 2026-03-31T21:28:01.418 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179067.log 2026-03-31T21:28:01.419 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179018.log.gz 2026-03-31T21:28:01.419 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179116.log 2026-03-31T21:28:01.419 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179067.log.gz 2026-03-31T21:28:01.419 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179165.log 2026-03-31T21:28:01.420 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179116.log.gz 2026-03-31T21:28:01.420 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179214.log 2026-03-31T21:28:01.421 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179165.log.gz 2026-03-31T21:28:01.421 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179263.log 2026-03-31T21:28:01.421 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179214.log.gz 2026-03-31T21:28:01.422 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179312.log 2026-03-31T21:28:01.422 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179263.log.gz 2026-03-31T21:28:01.422 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179361.log 2026-03-31T21:28:01.423 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179312.log.gz 2026-03-31T21:28:01.423 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179410.log 2026-03-31T21:28:01.423 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179361.log.gz 2026-03-31T21:28:01.424 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179459.log 2026-03-31T21:28:01.424 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179410.log.gz 2026-03-31T21:28:01.424 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179508.log 2026-03-31T21:28:01.425 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179459.log: 29.7% -- replaced with /var/log/ceph/ceph-client.admin.179459.log.gz 2026-03-31T21:28:01.425 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179557.log 2026-03-31T21:28:01.425 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179508.log.gz 2026-03-31T21:28:01.426 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179606.log 2026-03-31T21:28:01.426 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179557.log.gz 2026-03-31T21:28:01.426 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179655.log 2026-03-31T21:28:01.427 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179606.log.gz 2026-03-31T21:28:01.427 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179704.log 2026-03-31T21:28:01.427 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179655.log.gz 2026-03-31T21:28:01.428 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179753.log 2026-03-31T21:28:01.428 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179704.log.gz 2026-03-31T21:28:01.428 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179802.log 2026-03-31T21:28:01.429 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179753.log.gz 2026-03-31T21:28:01.429 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179851.log 2026-03-31T21:28:01.429 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179802.log.gz 2026-03-31T21:28:01.430 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179900.log 2026-03-31T21:28:01.430 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179851.log.gz 2026-03-31T21:28:01.430 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179949.log 2026-03-31T21:28:01.431 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179900.log.gz 2026-03-31T21:28:01.431 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.179998.log 2026-03-31T21:28:01.431 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179949.log.gz 2026-03-31T21:28:01.432 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180047.log 2026-03-31T21:28:01.432 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.179998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.179998.log.gz 2026-03-31T21:28:01.432 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180096.log 2026-03-31T21:28:01.433 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180047.log.gz 2026-03-31T21:28:01.433 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180145.log 2026-03-31T21:28:01.433 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180096.log.gz 2026-03-31T21:28:01.434 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180194.log 2026-03-31T21:28:01.434 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180145.log.gz 2026-03-31T21:28:01.434 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180243.log 2026-03-31T21:28:01.435 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180194.log.gz 2026-03-31T21:28:01.435 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180292.log 2026-03-31T21:28:01.435 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180243.log.gz 2026-03-31T21:28:01.436 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180341.log 2026-03-31T21:28:01.436 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180292.log.gz 2026-03-31T21:28:01.436 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180390.log 2026-03-31T21:28:01.437 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180341.log.gz 2026-03-31T21:28:01.437 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180439.log 2026-03-31T21:28:01.437 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180390.log.gz 2026-03-31T21:28:01.438 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180488.log 2026-03-31T21:28:01.438 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180439.log.gz 2026-03-31T21:28:01.438 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180537.log 2026-03-31T21:28:01.439 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180488.log.gz 2026-03-31T21:28:01.439 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180586.log 2026-03-31T21:28:01.439 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180537.log.gz 2026-03-31T21:28:01.440 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180636.log 2026-03-31T21:28:01.440 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180586.log.gz 2026-03-31T21:28:01.440 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180685.log 2026-03-31T21:28:01.441 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180636.log.gz 2026-03-31T21:28:01.441 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180734.log 2026-03-31T21:28:01.442 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180685.log.gz 2026-03-31T21:28:01.442 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180783.log 2026-03-31T21:28:01.442 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180734.log.gz 2026-03-31T21:28:01.442 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180832.log 2026-03-31T21:28:01.443 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180783.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180783.log.gz 2026-03-31T21:28:01.443 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180881.log 2026-03-31T21:28:01.444 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180832.log.gz 2026-03-31T21:28:01.444 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180930.log 2026-03-31T21:28:01.444 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180881.log.gz 2026-03-31T21:28:01.445 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.180979.log 2026-03-31T21:28:01.445 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180930.log.gz 2026-03-31T21:28:01.445 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181028.log 2026-03-31T21:28:01.446 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.180979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.180979.log.gz 2026-03-31T21:28:01.446 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181077.log 2026-03-31T21:28:01.446 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181028.log.gz 2026-03-31T21:28:01.447 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181126.log 2026-03-31T21:28:01.447 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181077.log.gz 2026-03-31T21:28:01.447 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181175.log 2026-03-31T21:28:01.448 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181126.log.gz 2026-03-31T21:28:01.448 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181224.log 2026-03-31T21:28:01.448 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181175.log.gz 2026-03-31T21:28:01.449 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181273.log 2026-03-31T21:28:01.449 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181224.log.gz 2026-03-31T21:28:01.449 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181322.log 2026-03-31T21:28:01.450 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181273.log.gz 2026-03-31T21:28:01.450 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181371.log 2026-03-31T21:28:01.451 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181322.log.gz 2026-03-31T21:28:01.451 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181420.log 2026-03-31T21:28:01.451 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181371.log.gz 2026-03-31T21:28:01.452 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181469.log 2026-03-31T21:28:01.452 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181420.log.gz 2026-03-31T21:28:01.452 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181518.log 2026-03-31T21:28:01.453 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181469.log.gz 2026-03-31T21:28:01.453 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181567.log 2026-03-31T21:28:01.453 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181518.log.gz 2026-03-31T21:28:01.454 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181617.log 2026-03-31T21:28:01.454 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181567.log.gz 2026-03-31T21:28:01.454 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181666.log 2026-03-31T21:28:01.455 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181617.log.gz 2026-03-31T21:28:01.455 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181715.log 2026-03-31T21:28:01.455 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181666.log.gz 2026-03-31T21:28:01.455 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181764.log 2026-03-31T21:28:01.456 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181715.log.gz 2026-03-31T21:28:01.456 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181813.log 2026-03-31T21:28:01.456 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181764.log.gz 2026-03-31T21:28:01.457 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181862.log 2026-03-31T21:28:01.457 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181813.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181813.log.gz 2026-03-31T21:28:01.457 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181911.log 2026-03-31T21:28:01.458 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181862.log.gz 2026-03-31T21:28:01.458 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.181960.log 2026-03-31T21:28:01.459 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181911.log.gz 2026-03-31T21:28:01.459 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182009.log 2026-03-31T21:28:01.459 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.181960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.181960.log.gz 2026-03-31T21:28:01.460 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182058.log 2026-03-31T21:28:01.460 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182009.log.gz 2026-03-31T21:28:01.460 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182108.log 2026-03-31T21:28:01.461 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182058.log.gz 2026-03-31T21:28:01.461 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182158.log 2026-03-31T21:28:01.461 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182108.log.gz 2026-03-31T21:28:01.461 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182215.log 2026-03-31T21:28:01.462 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182158.log.gz 2026-03-31T21:28:01.462 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182264.log 2026-03-31T21:28:01.463 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182215.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182215.log.gz 2026-03-31T21:28:01.463 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182313.log 2026-03-31T21:28:01.463 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182264.log.gz 2026-03-31T21:28:01.464 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182362.log 2026-03-31T21:28:01.464 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182313.log.gz 2026-03-31T21:28:01.464 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182411.log 2026-03-31T21:28:01.465 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182362.log.gz 2026-03-31T21:28:01.465 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182460.log 2026-03-31T21:28:01.465 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182411.log.gz 2026-03-31T21:28:01.465 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182509.log 2026-03-31T21:28:01.466 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182460.log.gz 2026-03-31T21:28:01.466 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182558.log 2026-03-31T21:28:01.467 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182509.log.gz 2026-03-31T21:28:01.467 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182607.log 2026-03-31T21:28:01.467 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182558.log.gz 2026-03-31T21:28:01.468 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182656.log 2026-03-31T21:28:01.468 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182607.log.gz 2026-03-31T21:28:01.468 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182705.log 2026-03-31T21:28:01.469 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182656.log.gz 2026-03-31T21:28:01.469 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182754.log 2026-03-31T21:28:01.469 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182705.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182705.log.gz 2026-03-31T21:28:01.470 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182803.log 2026-03-31T21:28:01.470 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182754.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182754.log.gz 2026-03-31T21:28:01.470 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182852.log 2026-03-31T21:28:01.471 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182803.log.gz 2026-03-31T21:28:01.471 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182901.log 2026-03-31T21:28:01.471 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182852.log.gz 2026-03-31T21:28:01.472 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182950.log 2026-03-31T21:28:01.472 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182901.log: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.182901.log.gz 2026-03-31T21:28:01.472 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.182999.log 2026-03-31T21:28:01.473 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182950.log.gz 2026-03-31T21:28:01.473 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183048.log 2026-03-31T21:28:01.473 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.182999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.182999.log.gz 2026-03-31T21:28:01.474 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183097.log 2026-03-31T21:28:01.474 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183048.log.gz 2026-03-31T21:28:01.474 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183146.log 2026-03-31T21:28:01.475 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183097.log.gz 2026-03-31T21:28:01.475 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183195.log 2026-03-31T21:28:01.475 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183146.log.gz 2026-03-31T21:28:01.476 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183244.log 2026-03-31T21:28:01.476 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183195.log.gz 2026-03-31T21:28:01.476 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183293.log 2026-03-31T21:28:01.477 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183244.log.gz 2026-03-31T21:28:01.477 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183342.log 2026-03-31T21:28:01.477 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183293.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183293.log.gz 2026-03-31T21:28:01.477 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183391.log 2026-03-31T21:28:01.478 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183342.log.gz 2026-03-31T21:28:01.478 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183440.log 2026-03-31T21:28:01.479 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183391.log.gz 2026-03-31T21:28:01.479 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183489.log 2026-03-31T21:28:01.479 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183440.log.gz 2026-03-31T21:28:01.479 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183538.log 2026-03-31T21:28:01.480 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183489.log.gz 2026-03-31T21:28:01.480 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183587.log 2026-03-31T21:28:01.480 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183538.log.gz 2026-03-31T21:28:01.481 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183636.log 2026-03-31T21:28:01.481 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183587.log.gz 2026-03-31T21:28:01.481 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183685.log 2026-03-31T21:28:01.482 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183636.log.gz 2026-03-31T21:28:01.482 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183734.log 2026-03-31T21:28:01.482 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183685.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183685.log.gz 2026-03-31T21:28:01.483 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183783.log 2026-03-31T21:28:01.483 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183734.log.gz 2026-03-31T21:28:01.483 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183832.log 2026-03-31T21:28:01.484 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183783.log: 29.9% -- replaced with /var/log/ceph/ceph-client.admin.183783.log.gz 2026-03-31T21:28:01.484 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183881.log 2026-03-31T21:28:01.484 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183832.log.gz 2026-03-31T21:28:01.485 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183930.log 2026-03-31T21:28:01.485 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183881.log.gz 2026-03-31T21:28:01.485 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.183979.log 2026-03-31T21:28:01.486 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183930.log.gz 2026-03-31T21:28:01.486 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184028.log 2026-03-31T21:28:01.486 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.183979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.183979.log.gz 2026-03-31T21:28:01.487 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184077.log 2026-03-31T21:28:01.487 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184028.log.gz 2026-03-31T21:28:01.487 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184126.log 2026-03-31T21:28:01.488 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184077.log.gz 2026-03-31T21:28:01.488 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184175.log 2026-03-31T21:28:01.488 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184126.log.gz 2026-03-31T21:28:01.489 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184224.log 2026-03-31T21:28:01.489 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184175.log.gz 2026-03-31T21:28:01.490 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184273.log 2026-03-31T21:28:01.490 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184224.log.gz 2026-03-31T21:28:01.490 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184322.log 2026-03-31T21:28:01.491 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184273.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184273.log.gz 2026-03-31T21:28:01.491 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184371.log 2026-03-31T21:28:01.491 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184322.log.gz 2026-03-31T21:28:01.492 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184420.log 2026-03-31T21:28:01.492 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184371.log.gz 2026-03-31T21:28:01.492 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184469.log 2026-03-31T21:28:01.493 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184420.log.gz 2026-03-31T21:28:01.493 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184518.log 2026-03-31T21:28:01.493 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184469.log.gz 2026-03-31T21:28:01.493 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184567.log 2026-03-31T21:28:01.494 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184518.log.gz 2026-03-31T21:28:01.494 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184616.log 2026-03-31T21:28:01.495 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184567.log.gz 2026-03-31T21:28:01.495 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184665.log 2026-03-31T21:28:01.495 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184616.log.gz 2026-03-31T21:28:01.495 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184714.log 2026-03-31T21:28:01.496 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184665.log.gz 2026-03-31T21:28:01.496 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184763.log 2026-03-31T21:28:01.496 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184714.log.gz 2026-03-31T21:28:01.497 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184812.log 2026-03-31T21:28:01.497 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184763.log.gz 2026-03-31T21:28:01.497 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184861.log 2026-03-31T21:28:01.498 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184812.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184812.log.gz 2026-03-31T21:28:01.498 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184910.log 2026-03-31T21:28:01.498 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184861.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184861.log.gz 2026-03-31T21:28:01.499 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.184959.log 2026-03-31T21:28:01.499 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184910.log.gz 2026-03-31T21:28:01.499 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185008.log 2026-03-31T21:28:01.500 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.184959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.184959.log.gz 2026-03-31T21:28:01.500 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185057.log 2026-03-31T21:28:01.500 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185008.log.gz 2026-03-31T21:28:01.501 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185106.log 2026-03-31T21:28:01.501 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185057.log.gz 2026-03-31T21:28:01.501 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185155.log 2026-03-31T21:28:01.502 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185106.log.gz 2026-03-31T21:28:01.502 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185204.log 2026-03-31T21:28:01.502 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185155.log.gz 2026-03-31T21:28:01.503 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185253.log 2026-03-31T21:28:01.503 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185204.log.gz 2026-03-31T21:28:01.503 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185302.log 2026-03-31T21:28:01.504 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185253.log.gz 2026-03-31T21:28:01.504 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185351.log 2026-03-31T21:28:01.504 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185302.log.gz 2026-03-31T21:28:01.505 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185400.log 2026-03-31T21:28:01.505 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185351.log.gz 2026-03-31T21:28:01.506 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185449.log 2026-03-31T21:28:01.506 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185400.log.gz 2026-03-31T21:28:01.506 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185498.log 2026-03-31T21:28:01.507 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185449.log.gz 2026-03-31T21:28:01.507 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185547.log 2026-03-31T21:28:01.507 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185498.log.gz 2026-03-31T21:28:01.507 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185596.log 2026-03-31T21:28:01.508 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185547.log.gz 2026-03-31T21:28:01.508 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185645.log 2026-03-31T21:28:01.508 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185596.log.gz 2026-03-31T21:28:01.509 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185694.log 2026-03-31T21:28:01.509 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185645.log.gz 2026-03-31T21:28:01.509 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185743.log 2026-03-31T21:28:01.510 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185694.log.gz 2026-03-31T21:28:01.510 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185792.log 2026-03-31T21:28:01.510 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185743.log.gz 2026-03-31T21:28:01.511 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185841.log 2026-03-31T21:28:01.511 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185792.log.gz 2026-03-31T21:28:01.511 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185890.log 2026-03-31T21:28:01.512 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185841.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185841.log.gz 2026-03-31T21:28:01.512 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185940.log 2026-03-31T21:28:01.513 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185890.log.gz 2026-03-31T21:28:01.513 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.185989.log 2026-03-31T21:28:01.513 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185940.log.gz 2026-03-31T21:28:01.514 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186038.log 2026-03-31T21:28:01.514 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.185989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.185989.log.gz 2026-03-31T21:28:01.514 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186087.log 2026-03-31T21:28:01.515 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186038.log.gz 2026-03-31T21:28:01.515 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186136.log 2026-03-31T21:28:01.515 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186087.log.gz 2026-03-31T21:28:01.516 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186185.log 2026-03-31T21:28:01.516 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186136.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186136.log.gz 2026-03-31T21:28:01.516 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186234.log 2026-03-31T21:28:01.517 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186185.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186185.log.gz 2026-03-31T21:28:01.517 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186283.log 2026-03-31T21:28:01.517 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186234.log.gz 2026-03-31T21:28:01.517 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186332.log 2026-03-31T21:28:01.518 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186283.log.gz 2026-03-31T21:28:01.518 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186381.log 2026-03-31T21:28:01.518 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186332.log.gz 2026-03-31T21:28:01.519 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186430.log 2026-03-31T21:28:01.519 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186381.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186381.log.gz 2026-03-31T21:28:01.519 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186479.log 2026-03-31T21:28:01.520 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186430.log.gz 2026-03-31T21:28:01.520 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186528.log 2026-03-31T21:28:01.520 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186479.log.gz 2026-03-31T21:28:01.521 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186577.log 2026-03-31T21:28:01.521 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186528.log.gz 2026-03-31T21:28:01.521 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186626.log 2026-03-31T21:28:01.522 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186577.log.gz 2026-03-31T21:28:01.522 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186675.log 2026-03-31T21:28:01.522 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186626.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186626.log.gz 2026-03-31T21:28:01.523 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186724.log 2026-03-31T21:28:01.523 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186675.log.gz 2026-03-31T21:28:01.523 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186773.log 2026-03-31T21:28:01.524 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186724.log.gz 2026-03-31T21:28:01.524 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186822.log 2026-03-31T21:28:01.525 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186773.log.gz 2026-03-31T21:28:01.525 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186871.log 2026-03-31T21:28:01.525 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186822.log.gz 2026-03-31T21:28:01.525 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186920.log 2026-03-31T21:28:01.526 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186871.log.gz 2026-03-31T21:28:01.526 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.186969.log 2026-03-31T21:28:01.527 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186920.log.gz 2026-03-31T21:28:01.527 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187018.log 2026-03-31T21:28:01.527 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.186969.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.186969.log.gz 2026-03-31T21:28:01.528 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187067.log 2026-03-31T21:28:01.528 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187018.log.gz 2026-03-31T21:28:01.528 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187116.log 2026-03-31T21:28:01.529 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187067.log.gz 2026-03-31T21:28:01.529 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187165.log 2026-03-31T21:28:01.529 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187116.log.gz 2026-03-31T21:28:01.530 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187214.log 2026-03-31T21:28:01.530 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187165.log.gz 2026-03-31T21:28:01.530 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187263.log 2026-03-31T21:28:01.531 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187214.log.gz 2026-03-31T21:28:01.531 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187312.log 2026-03-31T21:28:01.531 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187263.log.gz 2026-03-31T21:28:01.532 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187361.log 2026-03-31T21:28:01.532 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187312.log.gz 2026-03-31T21:28:01.532 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187410.log 2026-03-31T21:28:01.533 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187361.log.gz 2026-03-31T21:28:01.533 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187459.log 2026-03-31T21:28:01.533 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187410.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187410.log.gz 2026-03-31T21:28:01.534 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187508.log 2026-03-31T21:28:01.534 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187459.log.gz 2026-03-31T21:28:01.534 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187557.log 2026-03-31T21:28:01.535 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187508.log.gz 2026-03-31T21:28:01.535 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187606.log 2026-03-31T21:28:01.536 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187557.log.gz 2026-03-31T21:28:01.536 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187655.log 2026-03-31T21:28:01.536 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187606.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187606.log.gz 2026-03-31T21:28:01.537 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187704.log 2026-03-31T21:28:01.537 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187655.log: 30.2% -- replaced with /var/log/ceph/ceph-client.admin.187655.log.gz 2026-03-31T21:28:01.537 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187753.log 2026-03-31T21:28:01.538 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187704.log.gz 2026-03-31T21:28:01.538 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187802.log 2026-03-31T21:28:01.539 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187753.log.gz 2026-03-31T21:28:01.539 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187851.log 2026-03-31T21:28:01.539 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187802.log.gz 2026-03-31T21:28:01.540 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187900.log 2026-03-31T21:28:01.540 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187851.log.gz 2026-03-31T21:28:01.540 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187949.log 2026-03-31T21:28:01.541 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187900.log.gz 2026-03-31T21:28:01.541 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.187998.log 2026-03-31T21:28:01.541 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187949.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187949.log.gz 2026-03-31T21:28:01.542 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188047.log 2026-03-31T21:28:01.542 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.187998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.187998.log.gz 2026-03-31T21:28:01.543 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188096.log 2026-03-31T21:28:01.543 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188047.log.gz 2026-03-31T21:28:01.543 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188145.log 2026-03-31T21:28:01.544 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188096.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188096.log.gz 2026-03-31T21:28:01.544 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188194.log 2026-03-31T21:28:01.544 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188145.log.gz 2026-03-31T21:28:01.545 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188243.log 2026-03-31T21:28:01.545 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188194.log.gz 2026-03-31T21:28:01.545 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188292.log 2026-03-31T21:28:01.546 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188243.log.gz 2026-03-31T21:28:01.546 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188341.log 2026-03-31T21:28:01.547 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188292.log.gz 2026-03-31T21:28:01.547 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188390.log 2026-03-31T21:28:01.547 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188341.log.gz 2026-03-31T21:28:01.548 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188439.log 2026-03-31T21:28:01.548 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188390.log.gz 2026-03-31T21:28:01.548 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188488.log 2026-03-31T21:28:01.549 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188439.log: 29.7% -- replaced with /var/log/ceph/ceph-client.admin.188439.log.gz 2026-03-31T21:28:01.549 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188537.log 2026-03-31T21:28:01.549 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188488.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188488.log.gz 2026-03-31T21:28:01.550 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188586.log 2026-03-31T21:28:01.550 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188537.log.gz 2026-03-31T21:28:01.550 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188635.log 2026-03-31T21:28:01.551 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188586.log.gz 2026-03-31T21:28:01.551 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188684.log 2026-03-31T21:28:01.552 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188635.log.gz 2026-03-31T21:28:01.552 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188733.log 2026-03-31T21:28:01.552 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188684.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188684.log.gz 2026-03-31T21:28:01.553 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188782.log 2026-03-31T21:28:01.553 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188733.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188733.log.gz 2026-03-31T21:28:01.553 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188831.log 2026-03-31T21:28:01.554 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188782.log.gz 2026-03-31T21:28:01.554 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188880.log 2026-03-31T21:28:01.555 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188831.log.gz 2026-03-31T21:28:01.555 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188929.log 2026-03-31T21:28:01.555 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188880.log.gz 2026-03-31T21:28:01.555 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.188978.log 2026-03-31T21:28:01.556 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188929.log.gz 2026-03-31T21:28:01.556 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189027.log 2026-03-31T21:28:01.557 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.188978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.188978.log.gz 2026-03-31T21:28:01.557 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189076.log 2026-03-31T21:28:01.557 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189027.log.gz 2026-03-31T21:28:01.558 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189125.log 2026-03-31T21:28:01.558 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189076.log.gz 2026-03-31T21:28:01.558 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189174.log 2026-03-31T21:28:01.559 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189125.log.gz 2026-03-31T21:28:01.559 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189223.log 2026-03-31T21:28:01.560 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189174.log.gz 2026-03-31T21:28:01.560 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189272.log 2026-03-31T21:28:01.560 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189223.log.gz 2026-03-31T21:28:01.561 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189321.log 2026-03-31T21:28:01.561 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189272.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189272.log.gz 2026-03-31T21:28:01.561 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189370.log 2026-03-31T21:28:01.562 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189321.log.gz 2026-03-31T21:28:01.562 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189419.log 2026-03-31T21:28:01.562 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189370.log.gz 2026-03-31T21:28:01.563 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189468.log 2026-03-31T21:28:01.563 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189419.log.gz 2026-03-31T21:28:01.563 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189517.log 2026-03-31T21:28:01.564 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189468.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189468.log.gz 2026-03-31T21:28:01.564 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189566.log 2026-03-31T21:28:01.565 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189517.log.gz 2026-03-31T21:28:01.565 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189615.log 2026-03-31T21:28:01.565 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189566.log.gz 2026-03-31T21:28:01.566 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189664.log 2026-03-31T21:28:01.566 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189615.log.gz 2026-03-31T21:28:01.566 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189713.log 2026-03-31T21:28:01.567 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189664.log.gz 2026-03-31T21:28:01.567 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189762.log 2026-03-31T21:28:01.567 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189713.log.gz 2026-03-31T21:28:01.568 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189811.log 2026-03-31T21:28:01.568 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189762.log.gz 2026-03-31T21:28:01.568 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189860.log 2026-03-31T21:28:01.569 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189811.log.gz 2026-03-31T21:28:01.569 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189909.log 2026-03-31T21:28:01.570 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189860.log.gz 2026-03-31T21:28:01.570 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.189958.log 2026-03-31T21:28:01.570 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189909.log.gz 2026-03-31T21:28:01.571 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190007.log 2026-03-31T21:28:01.571 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.189958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.189958.log.gz 2026-03-31T21:28:01.571 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190057.log 2026-03-31T21:28:01.572 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190007.log.gz 2026-03-31T21:28:01.572 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190106.log 2026-03-31T21:28:01.572 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190057.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190057.log.gz 2026-03-31T21:28:01.573 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190155.log 2026-03-31T21:28:01.573 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190106.log.gz 2026-03-31T21:28:01.573 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190204.log 2026-03-31T21:28:01.574 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190155.log.gz 2026-03-31T21:28:01.574 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190253.log 2026-03-31T21:28:01.574 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190204.log.gz 2026-03-31T21:28:01.575 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190302.log 2026-03-31T21:28:01.575 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190253.log.gz 2026-03-31T21:28:01.576 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190351.log 2026-03-31T21:28:01.576 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190302.log.gz 2026-03-31T21:28:01.576 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190400.log 2026-03-31T21:28:01.577 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190351.log.gz 2026-03-31T21:28:01.577 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190449.log 2026-03-31T21:28:01.577 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190400.log.gz 2026-03-31T21:28:01.578 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190498.log 2026-03-31T21:28:01.578 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190449.log.gz 2026-03-31T21:28:01.579 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190548.log 2026-03-31T21:28:01.579 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190498.log.gz 2026-03-31T21:28:01.579 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190597.log 2026-03-31T21:28:01.580 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190548.log.gz 2026-03-31T21:28:01.580 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190646.log 2026-03-31T21:28:01.580 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190597.log.gz 2026-03-31T21:28:01.581 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190695.log 2026-03-31T21:28:01.581 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190646.log.gz 2026-03-31T21:28:01.581 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190744.log 2026-03-31T21:28:01.582 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190695.log.gz 2026-03-31T21:28:01.582 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190793.log 2026-03-31T21:28:01.582 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190744.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190744.log.gz 2026-03-31T21:28:01.583 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190842.log 2026-03-31T21:28:01.583 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190793.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190793.log.gz 2026-03-31T21:28:01.583 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190891.log 2026-03-31T21:28:01.584 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190842.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190842.log.gz 2026-03-31T21:28:01.584 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190940.log 2026-03-31T21:28:01.585 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190891.log.gz 2026-03-31T21:28:01.585 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.190989.log 2026-03-31T21:28:01.585 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190940.log.gz 2026-03-31T21:28:01.586 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191050.log 2026-03-31T21:28:01.586 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.190989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.190989.log.gz 2026-03-31T21:28:01.586 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.191097.log 2026-03-31T21:28:01.587 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.191050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.191050.log.gz 2026-03-31T21:28:01.587 INFO:teuthology.orchestra.run.vm00.stderr:/var/log/ceph/ceph-client.admin.191097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.191097.log.gz 2026-03-31T21:28:01.623 INFO:teuthology.orchestra.run.vm00.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-03-31T21:28:01.903 INFO:teuthology.orchestra.run.vm00.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-03-31T21:28:05.597 INFO:teuthology.orchestra.run.vm00.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-03-31T21:28:22.930 INFO:teuthology.orchestra.run.vm00.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-03-31T21:28:22.931 INFO:teuthology.orchestra.run.vm00.stderr: 2026-03-31T21:28:22.931 INFO:teuthology.orchestra.run.vm00.stderr:real 0m24.861s 2026-03-31T21:28:22.931 INFO:teuthology.orchestra.run.vm00.stderr:user 0m37.484s 2026-03-31T21:28:22.931 INFO:teuthology.orchestra.run.vm00.stderr:sys 0m1.942s 2026-03-31T21:28:22.931 INFO:tasks.ceph:Archiving logs... 2026-03-31T21:28:22.931 DEBUG:teuthology.misc:Transferring archived files from vm00:/var/log/ceph to /archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348/remote/vm00/log 2026-03-31T21:28:22.932 DEBUG:teuthology.orchestra.run.vm00:> sudo tar c -f - -C /var/log/ceph -- . 2026-03-31T21:28:25.207 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-03-31T21:28:25.209 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-03-31T21:28:25.209 DEBUG:teuthology.orchestra.run.vm00:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-03-31T21:28:25.257 INFO:teuthology.task.install.rpm:Removing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on rpm system. 2026-03-31T21:28:25.258 DEBUG:teuthology.orchestra.run.vm00:> 2026-03-31T21:28:25.258 DEBUG:teuthology.orchestra.run.vm00:> for d in ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ; do 2026-03-31T21:28:25.258 DEBUG:teuthology.orchestra.run.vm00:> sudo yum -y remove $d || true 2026-03-31T21:28:25.258 DEBUG:teuthology.orchestra.run.vm00:> done 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout: ceph-radosgw x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 103 M 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:Removing unused dependencies: 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout: mailcap noarch 2.1.49-5.el9 @baseos 78 k 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:25.540 INFO:teuthology.orchestra.run.vm00.stdout:Remove 2 Packages 2026-03-31T21:28:25.541 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:25.541 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 103 M 2026-03-31T21:28:25.541 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:25.545 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:25.545 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:25.562 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:25.563 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:25.596 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:25.617 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:25.617 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:25.617 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-03-31T21:28:25.617 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/ceph-radosgw.target". 2026-03-31T21:28:25.617 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/ceph.target.wants/ceph-radosgw.target". 2026-03-31T21:28:25.617 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:25.624 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:25.633 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:25.649 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : mailcap-2.1.49-5.el9.noarch 2/2 2026-03-31T21:28:26.367 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: mailcap-2.1.49-5.el9.noarch 2/2 2026-03-31T21:28:26.367 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:26.415 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 2/2 2026-03-31T21:28:26.415 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:26.415 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:26.415 INFO:teuthology.orchestra.run.vm00.stdout: ceph-radosgw-2:20.2.0-721.g5bb32787.el9.x86_64 mailcap-2.1.49-5.el9.noarch 2026-03-31T21:28:26.415 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:26.415 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout: ceph-test x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 362 M 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:Removing unused dependencies: 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout: socat x86_64 1.7.4.1-8.el9 @appstream 1.1 M 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout: xmlstarlet x86_64 1.6.1-20.el9 @appstream 195 k 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:Remove 3 Packages 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:26.602 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 363 M 2026-03-31T21:28:26.603 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:26.605 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:26.605 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:26.628 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:26.629 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:26.698 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:26.704 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-test-2:20.2.0-721.g5bb32787.el9.x86_64 1/3 2026-03-31T21:28:26.706 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : socat-1.7.4.1-8.el9.x86_64 2/3 2026-03-31T21:28:26.721 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : xmlstarlet-1.6.1-20.el9.x86_64 3/3 2026-03-31T21:28:26.784 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: xmlstarlet-1.6.1-20.el9.x86_64 3/3 2026-03-31T21:28:26.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-test-2:20.2.0-721.g5bb32787.el9.x86_64 1/3 2026-03-31T21:28:26.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 2/3 2026-03-31T21:28:26.834 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 3/3 2026-03-31T21:28:26.834 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:26.834 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:26.834 INFO:teuthology.orchestra.run.vm00.stdout: ceph-test-2:20.2.0-721.g5bb32787.el9.x86_64 socat-1.7.4.1-8.el9.x86_64 2026-03-31T21:28:26.834 INFO:teuthology.orchestra.run.vm00.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-03-31T21:28:26.834 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:26.834 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:27.020 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:27.020 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: ceph x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 0 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:Removing unused dependencies: 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mds x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 6.8 M 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mon x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 19 M 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: lua x86_64 5.4.4-4.el9 @appstream 593 k 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: lua-devel x86_64 5.4.4-4.el9 @crb 49 k 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: luarocks noarch 3.9.2-5.el9 @epel 692 k 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: unzip x86_64 6.0-59.el9 @baseos 389 k 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: zip x86_64 3.0-35.el9 @baseos 724 k 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:Remove 8 Packages 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 28 M 2026-03-31T21:28:27.021 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:27.023 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:27.023 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:27.050 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:27.050 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:27.091 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:27.096 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-2:20.2.0-721.g5bb32787.el9.x86_64 1/8 2026-03-31T21:28:27.099 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : luarocks-3.9.2-5.el9.noarch 2/8 2026-03-31T21:28:27.101 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : lua-devel-5.4.4-4.el9.x86_64 3/8 2026-03-31T21:28:27.103 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : zip-3.0-35.el9.x86_64 4/8 2026-03-31T21:28:27.105 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : unzip-6.0-59.el9.x86_64 5/8 2026-03-31T21:28:27.107 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : lua-5.4.4-4.el9.x86_64 6/8 2026-03-31T21:28:27.125 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 7/8 2026-03-31T21:28:27.125 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:27.125 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-03-31T21:28:27.125 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/ceph-mds.target". 2026-03-31T21:28:27.125 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/ceph.target.wants/ceph-mds.target". 2026-03-31T21:28:27.125 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.125 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 7/8 2026-03-31T21:28:27.132 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 7/8 2026-03-31T21:28:27.150 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 8/8 2026-03-31T21:28:27.150 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:27.150 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-03-31T21:28:27.150 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/ceph-mon.target". 2026-03-31T21:28:27.150 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/ceph.target.wants/ceph-mon.target". 2026-03-31T21:28:27.150 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.152 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 8/8 2026-03-31T21:28:27.270 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 8/8 2026-03-31T21:28:27.270 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-2:20.2.0-721.g5bb32787.el9.x86_64 1/8 2026-03-31T21:28:27.270 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 2/8 2026-03-31T21:28:27.271 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 3/8 2026-03-31T21:28:27.271 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : lua-5.4.4-4.el9.x86_64 4/8 2026-03-31T21:28:27.271 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 5/8 2026-03-31T21:28:27.271 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 6/8 2026-03-31T21:28:27.271 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : unzip-6.0-59.el9.x86_64 7/8 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : zip-3.0-35.el9.x86_64 8/8 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: ceph-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mds-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mon-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: lua-5.4.4-4.el9.x86_64 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: luarocks-3.9.2-5.el9.noarch 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: unzip-6.0-59.el9.x86_64 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: zip-3.0-35.el9.x86_64 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.314 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:27.516 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout:=========================================================================================== 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout:=========================================================================================== 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-base x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 24 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout:Removing dependent packages: 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-immutable-object-cache x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 447 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 2.9 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-cephadm noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 940 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-dashboard noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 140 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-diskprediction-local noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 66 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-rook noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 567 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-osd x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 54 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-volume noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 1.4 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: rbd-mirror x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 11 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout:Removing unused dependencies: 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: abseil-cpp x86_64 20211102.0-4.el9 @epel 1.9 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-common x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 98 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-grafana-dashboards noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 996 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-k8sevents noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 60 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-modules-core noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 1.6 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-prometheus-alerts noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 59 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ceph-selinux x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 138 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: cryptsetup x86_64 2.8.1-3.el9 @baseos 770 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas x86_64 3.0.4-9.el9 @appstream 68 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 @appstream 11 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 @appstream 39 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: gperftools-libs x86_64 2.9.1-3.el9 @epel 1.4 M 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: grpc-data noarch 1.46.7-10.el9 @epel 13 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: ledmon-libs x86_64 1.1.0-3.el9 @baseos 80 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: libcephsqlite x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 409 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: libconfig x86_64 1.7.2-9.el9 @baseos 220 k 2026-03-31T21:28:27.523 INFO:teuthology.orchestra.run.vm00.stdout: libgfortran x86_64 11.5.0-14.el9 @baseos 2.8 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: liboath x86_64 2.6.12-1.el9 @epel 94 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: libquadmath x86_64 11.5.0-14.el9 @baseos 330 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: libradosstriper1 x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 792 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: libstoragemgmt x86_64 1.10.1-1.el9 @appstream 685 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: libunwind x86_64 1.6.2-1.el9 @epel 170 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: libxslt x86_64 1.1.34-12.el9 @appstream 743 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: nvme-cli x86_64 2.16-1.el9 @baseos 7.0 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: openblas x86_64 0.3.29-1.el9 @appstream 112 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: openblas-openmp x86_64 0.3.29-1.el9 @appstream 46 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: pciutils x86_64 3.7.0-7.el9 @baseos 216 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: protobuf x86_64 3.14.0-17.el9 @appstream 3.5 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: protobuf-compiler x86_64 3.14.0-17.el9 @crb 2.9 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-asyncssh noarch 2.13.2-5.el9 @epel 3.9 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-autocommand noarch 2.2.2-8.el9 @epel 82 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-babel noarch 2.9.1-2.el9 @appstream 27 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 @epel 254 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-bcrypt x86_64 3.2.2-1.el9 @epel 87 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-cachetools noarch 4.2.4-1.el9 @epel 93 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-common x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 855 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-certifi noarch 2023.05.07-4.el9 @epel 6.3 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-cffi x86_64 1.14.5-5.el9 @baseos 1.0 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-chardet noarch 4.0.0-5.el9 @anaconda 1.4 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-cheroot noarch 10.0.1-5.el9 @epel 682 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-cherrypy noarch 18.10.0-5.el9 @epel 1.0 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-cryptography x86_64 36.0.1-5.el9 @baseos 4.5 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-devel x86_64 3.9.25-3.el9 @appstream 765 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-google-auth noarch 1:2.45.0-1.el9 @epel 1.4 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio x86_64 1.46.7-10.el9 @epel 6.7 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 @epel 418 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-idna noarch 2.10-7.el9.1 @anaconda 513 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-influxdb noarch 5.3.1-1.el9 @epel 747 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-isodate noarch 0.6.1-3.el9 @epel 203 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco noarch 8.2.1-3.el9 @epel 3.7 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 @epel 24 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 @epel 55 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-context noarch 6.0.1-3.el9 @epel 31 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 @epel 33 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-text noarch 4.0.0-2.el9 @epel 51 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jinja2 noarch 2.11.3-8.el9 @appstream 1.1 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jsonpatch noarch 1.21-16.el9 @koji-override-0 55 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-jsonpointer noarch 2.0-4.el9 @koji-override-0 34 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 @epel 21 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 @appstream 832 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-lxml x86_64 4.6.5-3.el9 @appstream 4.2 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-markupsafe x86_64 1.1.1-12.el9 @appstream 60 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-more-itertools noarch 8.12.0-2.el9 @epel 378 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-msgpack x86_64 1.0.3-2.el9 @epel 264 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-natsort noarch 7.1.1-5.el9 @epel 215 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy x86_64 1:1.23.5-2.el9 @appstream 30 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 @appstream 1.7 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-oauthlib noarch 3.1.1-5.el9 @koji-override-0 888 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-packaging noarch 20.9-5.el9 @appstream 248 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-ply noarch 3.11-14.el9 @baseos 430 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-portend noarch 3.1.0-2.el9 @epel 20 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-prettytable noarch 0.7.2-27.el9 @koji-override-0 166 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-protobuf noarch 3.14.0-17.el9 @appstream 1.4 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 @epel 389 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1 noarch 0.4.8-7.el9 @appstream 622 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 @appstream 1.0 M 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-pycparser noarch 2.20-6.el9 @baseos 745 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyparsing noarch 2.4.7-9.el9 @baseos 635 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-pysocks noarch 1.7.1-12.el9 @anaconda 88 k 2026-03-31T21:28:27.524 INFO:teuthology.orchestra.run.vm00.stdout: python3-pytz noarch 2021.1-5.el9 @koji-override-0 176 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-repoze-lru noarch 0.7-16.el9 @epel 83 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests noarch 2.25.1-10.el9 @baseos 405 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 @appstream 119 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-routes noarch 2.5.1-5.el9 @epel 459 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-rsa noarch 4.9-2.el9 @epel 202 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-saml noarch 1.16.0-1.el9 @epel 730 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-scipy x86_64 1.9.3-2.el9 @appstream 76 M 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-tempora noarch 5.0.0-2.el9 @epel 96 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-toml noarch 0.10.2-6.el9 @appstream 99 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-typing-extensions noarch 4.15.0-1.el9 @epel 447 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-urllib3 noarch 1.26.5-7.el9 @baseos 746 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-websocket-client noarch 1.2.3-2.el9 @epel 319 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-xmlsec x86_64 1.3.13-1.el9 @epel 158 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: python3-zc-lockfile noarch 2.0-10.el9 @epel 35 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: qatlib x86_64 25.08.0-2.el9 @appstream 639 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: qatlib-service x86_64 25.08.0-2.el9 @appstream 69 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: qatzip-libs x86_64 1.3.1-1.el9 @appstream 148 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: smartmontools x86_64 1:7.2-10.el9 @baseos 1.9 M 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1 x86_64 1.2.29-13.el9 @appstream 596 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 @appstream 281 k 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout:=========================================================================================== 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout:Remove 110 Packages 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 683 M 2026-03-31T21:28:27.525 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:27.553 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:27.553 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:27.675 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:27.675 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:27.843 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:27.843 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 1/110 2026-03-31T21:28:27.851 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 1/110 2026-03-31T21:28:27.868 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 2/110 2026-03-31T21:28:27.868 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:27.868 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-03-31T21:28:27.869 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/ceph-mgr.target". 2026-03-31T21:28:27.869 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/ceph.target.wants/ceph-mgr.target". 2026-03-31T21:28:27.869 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:27.869 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 2/110 2026-03-31T21:28:27.883 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 2/110 2026-03-31T21:28:27.944 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mgr-modules-core-2:20.2.0-721.g5bb32787.el9 3/110 2026-03-31T21:28:27.944 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.no 4/110 2026-03-31T21:28:27.961 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.no 4/110 2026-03-31T21:28:27.966 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-influxdb-5.3.1-1.el9.noarch 5/110 2026-03-31T21:28:27.966 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noar 6/110 2026-03-31T21:28:27.979 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noar 6/110 2026-03-31T21:28:27.987 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-cherrypy-18.10.0-5.el9.noarch 7/110 2026-03-31T21:28:27.992 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-cheroot-10.0.1-5.el9.noarch 8/110 2026-03-31T21:28:28.001 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/110 2026-03-31T21:28:28.004 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-grpcio-1.46.7-10.el9.x86_64 10/110 2026-03-31T21:28:28.026 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 11/110 2026-03-31T21:28:28.026 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:28.026 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-03-31T21:28:28.026 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/ceph-osd.target". 2026-03-31T21:28:28.026 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/ceph.target.wants/ceph-osd.target". 2026-03-31T21:28:28.026 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:28.030 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 11/110 2026-03-31T21:28:28.039 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 11/110 2026-03-31T21:28:28.056 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 12/110 2026-03-31T21:28:28.056 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:28.056 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-03-31T21:28:28.056 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:28.063 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 12/110 2026-03-31T21:28:28.073 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 12/110 2026-03-31T21:28:28.076 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jaraco-collections-3.0.0-8.el9.noarch 13/110 2026-03-31T21:28:28.080 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jaraco-text-4.0.0-2.el9.noarch 14/110 2026-03-31T21:28:28.085 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jinja2-2.11.3-8.el9.noarch 15/110 2026-03-31T21:28:28.112 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-saml-1.16.0-1.el9.noarch 16/110 2026-03-31T21:28:28.119 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-babel-2.9.1-2.el9.noarch 17/110 2026-03-31T21:28:28.123 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jaraco-classes-3.2.1-5.el9.noarch 18/110 2026-03-31T21:28:28.131 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-pyOpenSSL-21.0.0-1.el9.noarch 19/110 2026-03-31T21:28:28.141 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-asyncssh-2.13.2-5.el9.noarch 20/110 2026-03-31T21:28:28.141 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb3 21/110 2026-03-31T21:28:28.149 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb3 21/110 2026-03-31T21:28:28.244 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jsonpatch-1.21-16.el9.noarch 22/110 2026-03-31T21:28:28.260 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-scipy-1.9.3-2.el9.x86_64 23/110 2026-03-31T21:28:28.267 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-xmlsec-1.3.13-1.el9.x86_64 24/110 2026-03-31T21:28:28.271 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-lxml-4.6.5-3.el9.x86_64 25/110 2026-03-31T21:28:28.286 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 26/110 2026-03-31T21:28:28.286 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/libstoragemgmt.service". 2026-03-31T21:28:28.286 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:28.287 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libstoragemgmt-1.10.1-1.el9.x86_64 26/110 2026-03-31T21:28:28.316 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 26/110 2026-03-31T21:28:28.321 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 27/110 2026-03-31T21:28:28.323 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : xmlsec1-openssl-1.2.29-13.el9.x86_64 28/110 2026-03-31T21:28:28.336 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : xmlsec1-1.2.29-13.el9.x86_64 29/110 2026-03-31T21:28:28.341 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-cryptography-36.0.1-5.el9.x86_64 30/110 2026-03-31T21:28:28.343 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : protobuf-compiler-3.14.0-17.el9.x86_64 31/110 2026-03-31T21:28:28.346 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-bcrypt-3.2.2-1.el9.x86_64 32/110 2026-03-31T21:28:28.365 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 33/110 2026-03-31T21:28:28.365 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:28.365 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-03-31T21:28:28.365 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target". 2026-03-31T21:28:28.365 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target". 2026-03-31T21:28:28.365 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:28.367 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 33/110 2026-03-31T21:28:28.376 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 33/110 2026-03-31T21:28:28.379 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jaraco-context-6.0.1-3.el9.noarch 34/110 2026-03-31T21:28:28.382 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-packaging-20.9-5.el9.noarch 35/110 2026-03-31T21:28:28.385 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-portend-3.1.0-2.el9.noarch 36/110 2026-03-31T21:28:28.387 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-tempora-5.0.0-2.el9.noarch 37/110 2026-03-31T21:28:28.390 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jaraco-functools-3.5.0-2.el9.noarch 38/110 2026-03-31T21:28:28.393 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-routes-2.5.1-5.el9.noarch 39/110 2026-03-31T21:28:28.393 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.no 40/110 2026-03-31T21:28:28.443 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.no 40/110 2026-03-31T21:28:28.453 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-kubernetes-1:26.1.0-3.el9.noarch 41/110 2026-03-31T21:28:28.457 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-requests-oauthlib-1.3.0-12.el9.noarch 42/110 2026-03-31T21:28:28.465 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-requests-2.25.1-10.el9.noarch 43/110 2026-03-31T21:28:28.470 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-google-auth-1:2.45.0-1.el9.noarch 44/110 2026-03-31T21:28:28.480 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-rsa-4.9-2.el9.noarch 45/110 2026-03-31T21:28:28.487 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-pyasn1-modules-0.4.8-7.el9.noarch 46/110 2026-03-31T21:28:28.492 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-urllib3-1.26.5-7.el9.noarch 47/110 2026-03-31T21:28:28.498 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-cffi-1.14.5-5.el9.x86_64 48/110 2026-03-31T21:28:28.547 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-pycparser-2.20-6.el9.noarch 49/110 2026-03-31T21:28:28.558 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-numpy-1:1.23.5-2.el9.x86_64 50/110 2026-03-31T21:28:28.561 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : flexiblas-netlib-3.0.4-9.el9.x86_64 51/110 2026-03-31T21:28:28.565 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 52/110 2026-03-31T21:28:28.567 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : openblas-openmp-0.3.29-1.el9.x86_64 53/110 2026-03-31T21:28:28.571 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libgfortran-11.5.0-14.el9.x86_64 54/110 2026-03-31T21:28:28.574 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 55/110 2026-03-31T21:28:28.593 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-721.g5bb327 56/110 2026-03-31T21:28:28.593 INFO:teuthology.orchestra.run.vm00.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-03-31T21:28:28.593 INFO:teuthology.orchestra.run.vm00.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-03-31T21:28:28.593 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:28.594 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-immutable-object-cache-2:20.2.0-721.g5bb327 56/110 2026-03-31T21:28:28.603 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.2.0-721.g5bb327 56/110 2026-03-31T21:28:28.605 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : openblas-0.3.29-1.el9.x86_64 57/110 2026-03-31T21:28:28.608 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : flexiblas-3.0.4-9.el9.x86_64 58/110 2026-03-31T21:28:28.611 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-ply-3.11-14.el9.noarch 59/110 2026-03-31T21:28:28.615 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-idna-2.10-7.el9.1.noarch 60/110 2026-03-31T21:28:28.620 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-pysocks-1.7.1-12.el9.noarch 61/110 2026-03-31T21:28:28.625 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-pyasn1-0.4.8-7.el9.noarch 62/110 2026-03-31T21:28:28.631 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-cachetools-4.2.4-1.el9.noarch 63/110 2026-03-31T21:28:28.640 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-chardet-4.0.0-5.el9.noarch 64/110 2026-03-31T21:28:28.646 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-oauthlib-3.1.1-5.el9.noarch 65/110 2026-03-31T21:28:28.650 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-websocket-client-1.2.3-2.el9.noarch 66/110 2026-03-31T21:28:28.652 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-certifi-2023.05.07-4.el9.noarch 67/110 2026-03-31T21:28:28.654 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-repoze-lru-0.7-16.el9.noarch 68/110 2026-03-31T21:28:28.657 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jaraco-8.2.1-3.el9.noarch 69/110 2026-03-31T21:28:28.659 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-more-itertools-8.12.0-2.el9.noarch 70/110 2026-03-31T21:28:28.662 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-toml-0.10.2-6.el9.noarch 71/110 2026-03-31T21:28:28.665 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-pytz-2021.1-5.el9.noarch 72/110 2026-03-31T21:28:28.668 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-pyparsing-2.4.7-9.el9.noarch 73/110 2026-03-31T21:28:28.675 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-backports-tarfile-1.2.0-1.el9.noarch 74/110 2026-03-31T21:28:28.679 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-devel-3.9.25-3.el9.x86_64 75/110 2026-03-31T21:28:28.682 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-jsonpointer-2.0-4.el9.noarch 76/110 2026-03-31T21:28:28.685 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-typing-extensions-4.15.0-1.el9.noarch 77/110 2026-03-31T21:28:28.688 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-isodate-0.6.1-3.el9.noarch 78/110 2026-03-31T21:28:28.690 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-autocommand-2.2.2-8.el9.noarch 79/110 2026-03-31T21:28:28.695 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : grpc-data-1.46.7-10.el9.noarch 80/110 2026-03-31T21:28:28.699 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-protobuf-3.14.0-17.el9.noarch 81/110 2026-03-31T21:28:28.702 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-zc-lockfile-2.0-10.el9.noarch 82/110 2026-03-31T21:28:28.705 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-natsort-7.1.1-5.el9.noarch 83/110 2026-03-31T21:28:28.706 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-grafana-dashboards-2:20.2.0-721.g5bb32787.e 84/110 2026-03-31T21:28:28.708 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-prometheus-alerts-2:20.2.0-721.g5bb32787.el 85/110 2026-03-31T21:28:28.727 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 86/110 2026-03-31T21:28:28.727 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2026-03-31T21:28:28.727 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/ceph.target.wants/ceph-crash.service". 2026-03-31T21:28:28.727 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:28.733 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 86/110 2026-03-31T21:28:28.733 INFO:teuthology.orchestra.run.vm00.stdout:warning: file /etc/logrotate.d/ceph: remove failed: No such file or directory 2026-03-31T21:28:28.733 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:28.757 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 86/110 2026-03-31T21:28:28.757 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 87/110 2026-03-31T21:28:28.889 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 87/110 2026-03-31T21:28:28.896 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : qatzip-libs-1.3.1-1.el9.x86_64 88/110 2026-03-31T21:28:28.899 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-ceph-common-2:20.2.0-721.g5bb32787.el9.x 89/110 2026-03-31T21:28:28.901 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-prettytable-0.7.2-27.el9.noarch 90/110 2026-03-31T21:28:28.901 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 91/110 2026-03-31T21:28:34.290 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 91/110 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /sys 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /proc 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /mnt 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /var/tmp 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /home 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /root 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout:skipping the directory /tmp 2026-03-31T21:28:34.291 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.301 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : qatlib-25.08.0-2.el9.x86_64 92/110 2026-03-31T21:28:34.318 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 93/110 2026-03-31T21:28:34.318 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : qatlib-service-25.08.0-2.el9.x86_64 93/110 2026-03-31T21:28:34.326 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 93/110 2026-03-31T21:28:34.329 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : gperftools-libs-2.9.1-3.el9.x86_64 94/110 2026-03-31T21:28:34.331 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libunwind-1.6.2-1.el9.x86_64 95/110 2026-03-31T21:28:34.334 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : pciutils-3.7.0-7.el9.x86_64 96/110 2026-03-31T21:28:34.336 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : liboath-2.6.12-1.el9.x86_64 97/110 2026-03-31T21:28:34.336 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_ 98/110 2026-03-31T21:28:34.366 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_ 98/110 2026-03-31T21:28:34.372 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : nvme-cli-2.16-1.el9.x86_64 99/110 2026-03-31T21:28:34.384 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: smartmontools-1:7.2-10.el9.x86_64 100/110 2026-03-31T21:28:34.384 INFO:teuthology.orchestra.run.vm00.stdout:Removed "/etc/systemd/system/multi-user.target.wants/smartd.service". 2026-03-31T21:28:34.384 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.386 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : smartmontools-1:7.2-10.el9.x86_64 100/110 2026-03-31T21:28:34.393 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: smartmontools-1:7.2-10.el9.x86_64 100/110 2026-03-31T21:28:34.396 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ledmon-libs-1.1.0-3.el9.x86_64 101/110 2026-03-31T21:28:34.398 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libquadmath-11.5.0-14.el9.x86_64 102/110 2026-03-31T21:28:34.400 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : protobuf-3.14.0-17.el9.x86_64 103/110 2026-03-31T21:28:34.403 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libxslt-1.1.34-12.el9.x86_64 104/110 2026-03-31T21:28:34.405 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libconfig-1.7.2-9.el9.x86_64 105/110 2026-03-31T21:28:34.410 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-markupsafe-1.1.1-12.el9.x86_64 106/110 2026-03-31T21:28:34.418 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : cryptsetup-2.8.1-3.el9.x86_64 107/110 2026-03-31T21:28:34.423 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : abseil-cpp-20211102.0-4.el9.x86_64 108/110 2026-03-31T21:28:34.425 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-msgpack-1.0.3-2.el9.x86_64 109/110 2026-03-31T21:28:34.425 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 110/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 110/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 1/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 2/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 3/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-grafana-dashboards-2:20.2.0-721.g5bb32787.e 4/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-immutable-object-cache-2:20.2.0-721.g5bb327 5/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 6/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noar 7/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.no 8/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb3 9/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.no 10/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-modules-core-2:20.2.0-721.g5bb32787.el9 11/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 12/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 13/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-prometheus-alerts-2:20.2.0-721.g5bb32787.el 14/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 15/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 16/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 17/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 18/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 19/110 2026-03-31T21:28:34.528 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 20/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 22/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 23/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 24/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 25/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 26/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 27/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 28/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_ 29/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 30/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 31/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 32/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 33/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 34/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 35/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 36/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 37/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 38/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 39/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 40/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 41/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 42/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 43/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 44/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-ceph-common-2:20.2.0-721.g5bb32787.el9.x 45/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 46/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 47/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-chardet-4.0.0-5.el9.noarch 48/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cheroot-10.0.1-5.el9.noarch 49/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cherrypy-18.10.0-5.el9.noarch 50/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 51/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 52/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 53/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 54/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 55/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-idna-2.10-7.el9.1.noarch 56/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 57/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 58/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 59/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 60/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 61/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 62/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 63/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 64/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 65/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 66/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 67/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 68/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 69/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 70/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 72/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 73/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 74/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 75/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 76/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 77/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 78/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-ply-3.11-14.el9.noarch 79/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 80/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 81/110 2026-03-31T21:28:34.529 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 82/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 83/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 84/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 85/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 87/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pysocks-1.7.1-12.el9.noarch 88/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 89/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 90/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 91/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 93/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 94/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 95/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 96/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 97/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 98/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 99/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 100/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 101/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 102/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 103/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 104/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 105/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 106/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 107/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : smartmontools-1:7.2-10.el9.x86_64 108/110 2026-03-31T21:28:34.530 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 109/110 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 110/110 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-base-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-grafana-dashboards-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-immutable-object-cache-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-cephadm-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-dashboard-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-diskprediction-local-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-k8sevents-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-modules-core-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-mgr-rook-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-osd-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-prometheus-alerts-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-selinux-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ceph-volume-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: grpc-data-1.46.7-10.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libcephsqlite-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libconfig-1.7.2-9.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: liboath-2.6.12-1.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libradosstriper1-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libunwind-1.6.2-1.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: libxslt-1.1.34-12.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: nvme-cli-2.16-1.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: openblas-0.3.29-1.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: pciutils-3.7.0-7.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: protobuf-3.14.0-17.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: python3-babel-2.9.1-2.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-03-31T21:28:34.615 INFO:teuthology.orchestra.run.vm00.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-common-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-chardet-4.0.0-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-cheroot-10.0.1-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-cherrypy-18.10.0-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-idna-2.10-7.el9.1.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jsonpatch-1.21-16.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-jsonpointer-2.0-4.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-packaging-20.9-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-ply-3.11-14.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-portend-3.1.0-2.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-prettytable-0.7.2-27.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-pycparser-2.20-6.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-pysocks-1.7.1-12.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-pytz-2021.1-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests-2.25.1-10.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-routes-2.5.1-5.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-rsa-4.9-2.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-saml-1.16.0-1.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-03-31T21:28:34.616 INFO:teuthology.orchestra.run.vm00.stdout: python3-toml-0.10.2-6.el9.noarch 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: qatlib-25.08.0-2.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: rbd-mirror-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: smartmontools-1:7.2-10.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.617 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:34.823 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout: cephadm noarch 2:20.2.0-721.g5bb32787.el9 @ceph-noarch 1.0 M 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:Remove 1 Package 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 1.0 M 2026-03-31T21:28:34.824 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:34.825 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:34.825 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:34.827 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:34.827 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:34.843 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:34.843 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : cephadm-2:20.2.0-721.g5bb32787.el9.noarch 1/1 2026-03-31T21:28:34.958 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: cephadm-2:20.2.0-721.g5bb32787.el9.noarch 1/1 2026-03-31T21:28:34.992 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : cephadm-2:20.2.0-721.g5bb32787.el9.noarch 1/1 2026-03-31T21:28:34.992 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.992 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:34.992 INFO:teuthology.orchestra.run.vm00.stdout: cephadm-2:20.2.0-721.g5bb32787.el9.noarch 2026-03-31T21:28:34.992 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:34.992 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:35.155 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: ceph-immutable-object-cache 2026-03-31T21:28:35.155 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:35.159 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:35.160 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:35.160 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:35.322 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: ceph-mgr 2026-03-31T21:28:35.322 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:35.326 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:35.326 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:35.326 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:35.492 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: ceph-mgr-dashboard 2026-03-31T21:28:35.492 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:35.495 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:35.496 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:35.496 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:35.668 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: ceph-mgr-diskprediction-local 2026-03-31T21:28:35.669 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:35.672 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:35.673 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:35.673 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:35.832 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: ceph-mgr-rook 2026-03-31T21:28:35.832 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:35.835 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:35.836 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:35.836 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:35.991 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: ceph-mgr-cephadm 2026-03-31T21:28:35.991 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:35.995 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:35.995 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:35.995 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:36.166 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:36.166 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:36.166 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:28:36.166 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:36.166 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:36.166 INFO:teuthology.orchestra.run.vm00.stdout: ceph-fuse x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 2.7 M 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout:Removing unused dependencies: 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout: fuse x86_64 2.9.9-17.el9 @baseos 214 k 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout:Remove 2 Packages 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 2.9 M 2026-03-31T21:28:36.167 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:36.169 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:36.169 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:36.181 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:36.182 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:36.210 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:36.214 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : ceph-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:36.227 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : fuse-2.9.9-17.el9.x86_64 2/2 2026-03-31T21:28:36.289 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: fuse-2.9.9-17.el9.x86_64 2/2 2026-03-31T21:28:36.289 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : ceph-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:36.327 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : fuse-2.9.9-17.el9.x86_64 2/2 2026-03-31T21:28:36.327 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.327 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:36.327 INFO:teuthology.orchestra.run.vm00.stdout: ceph-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 fuse-2.9.9-17.el9.x86_64 2026-03-31T21:28:36.327 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.327 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:36.507 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: ceph-volume 2026-03-31T21:28:36.507 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:36.510 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:36.511 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:36.511 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repo Size 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout: librados-devel x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 449 k 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:Removing dependent packages: 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-devel x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 155 k 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:Remove 2 Packages 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.682 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 604 k 2026-03-31T21:28:36.683 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:36.684 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:36.684 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:36.694 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:36.694 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:36.721 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:36.723 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libcephfs-devel-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:36.740 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : librados-devel-2:20.2.0-721.g5bb32787.el9.x86_64 2/2 2026-03-31T21:28:36.802 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librados-devel-2:20.2.0-721.g5bb32787.el9.x86_64 2/2 2026-03-31T21:28:36.802 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs-devel-2:20.2.0-721.g5bb32787.el9.x86_64 1/2 2026-03-31T21:28:36.842 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librados-devel-2:20.2.0-721.g5bb32787.el9.x86_64 2/2 2026-03-31T21:28:36.842 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.842 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:36.842 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-devel-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:36.842 INFO:teuthology.orchestra.run.vm00.stdout: librados-devel-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:36.842 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:36.842 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:37.041 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repo Size 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs2 x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 2.4 M 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout:Removing dependent packages: 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout: python3-cephfs x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 510 k 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout:Removing unused dependencies: 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-daemon x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 90 k 2026-03-31T21:28:37.042 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-proxy2 x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 52 k 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-argparse x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 187 k 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout:Remove 5 Packages 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 3.3 M 2026-03-31T21:28:37.043 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:37.044 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:37.044 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:37.056 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:37.056 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:37.083 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:37.085 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-cephfs-2:20.2.0-721.g5bb32787.el9.x86_64 1/5 2026-03-31T21:28:37.086 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-ceph-argparse-2:20.2.0-721.g5bb32787.el9.x86 2/5 2026-03-31T21:28:37.086 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_64 3/5 2026-03-31T21:28:37.098 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_64 3/5 2026-03-31T21:28:37.100 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libcephfs-daemon-2:20.2.0-721.g5bb32787.el9.x86_64 4/5 2026-03-31T21:28:37.100 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 5/5 2026-03-31T21:28:37.159 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 5/5 2026-03-31T21:28:37.159 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs-daemon-2:20.2.0-721.g5bb32787.el9.x86_64 1/5 2026-03-31T21:28:37.159 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_64 2/5 2026-03-31T21:28:37.159 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 3/5 2026-03-31T21:28:37.159 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-ceph-argparse-2:20.2.0-721.g5bb32787.el9.x86 4/5 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-cephfs-2:20.2.0-721.g5bb32787.el9.x86_64 5/5 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-daemon-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs-proxy2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: libcephfs2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: python3-ceph-argparse-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: python3-cephfs-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.208 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:37.380 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: libcephfs-devel 2026-03-31T21:28:37.380 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:37.382 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:37.383 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:37.383 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:37.560 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:37.561 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:37.561 INFO:teuthology.orchestra.run.vm00.stdout: Package Arch Version Repository Size 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout:Removing: 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: librados2 x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 12 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout:Removing dependent packages: 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: python3-rados x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 1.1 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: python3-rbd x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 1.1 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: python3-rgw x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 264 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: qemu-kvm-block-rbd x86_64 17:10.1.0-16.el9 @appstream 37 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: rbd-fuse x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 238 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: rbd-nbd x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 498 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout:Removing unused dependencies: 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: boost-program-options x86_64 1.75.0-13.el9 @appstream 276 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: libarrow x86_64 9.0.0-15.el9 @epel 18 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: libarrow-doc noarch 9.0.0-15.el9 @epel 122 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: libnbd x86_64 1.20.3-4.el9 @appstream 453 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: libpmemobj x86_64 1.12.1-1.el9 @appstream 383 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: librabbitmq x86_64 0.11.0-7.el9 @appstream 102 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: librbd1 x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 10 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: librdkafka x86_64 1.6.1-102.el9 @appstream 2.0 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: librgw2 x86_64 2:20.2.0-721.g5bb32787.el9 @ceph 28 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: lttng-ust x86_64 2.12.0-6.el9 @appstream 1.0 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: parquet-libs x86_64 9.0.0-15.el9 @epel 2.8 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: re2 x86_64 1:20211101-20.el9 @epel 472 k 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: thrift x86_64 0.15.0-4.el9 @epel 4.8 M 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout:Transaction Summary 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout:================================================================================ 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout:Remove 20 Packages 2026-03-31T21:28:37.562 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.563 INFO:teuthology.orchestra.run.vm00.stdout:Freed space: 84 M 2026-03-31T21:28:37.563 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction check 2026-03-31T21:28:37.566 INFO:teuthology.orchestra.run.vm00.stdout:Transaction check succeeded. 2026-03-31T21:28:37.566 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction test 2026-03-31T21:28:37.586 INFO:teuthology.orchestra.run.vm00.stdout:Transaction test succeeded. 2026-03-31T21:28:37.587 INFO:teuthology.orchestra.run.vm00.stdout:Running transaction 2026-03-31T21:28:37.628 INFO:teuthology.orchestra.run.vm00.stdout: Preparing : 1/1 2026-03-31T21:28:37.631 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : rbd-nbd-2:20.2.0-721.g5bb32787.el9.x86_64 1/20 2026-03-31T21:28:37.633 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : rbd-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 2/20 2026-03-31T21:28:37.636 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-rgw-2:20.2.0-721.g5bb32787.el9.x86_64 3/20 2026-03-31T21:28:37.636 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 4/20 2026-03-31T21:28:37.649 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 4/20 2026-03-31T21:28:37.651 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : parquet-libs-9.0.0-15.el9.x86_64 5/20 2026-03-31T21:28:37.653 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-rbd-2:20.2.0-721.g5bb32787.el9.x86_64 6/20 2026-03-31T21:28:37.654 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : python3-rados-2:20.2.0-721.g5bb32787.el9.x86_64 7/20 2026-03-31T21:28:37.656 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : qemu-kvm-block-rbd-17:10.1.0-16.el9.x86_64 8/20 2026-03-31T21:28:37.658 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libarrow-doc-9.0.0-15.el9.noarch 9/20 2026-03-31T21:28:37.658 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 10/20 2026-03-31T21:28:37.672 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 10/20 2026-03-31T21:28:37.672 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : librados2-2:20.2.0-721.g5bb32787.el9.x86_64 11/20 2026-03-31T21:28:37.673 INFO:teuthology.orchestra.run.vm00.stdout:warning: file /etc/ceph: remove failed: No such file or directory 2026-03-31T21:28:37.673 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.686 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librados2-2:20.2.0-721.g5bb32787.el9.x86_64 11/20 2026-03-31T21:28:37.689 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libarrow-9.0.0-15.el9.x86_64 12/20 2026-03-31T21:28:37.693 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : re2-1:20211101-20.el9.x86_64 13/20 2026-03-31T21:28:37.697 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : lttng-ust-2.12.0-6.el9.x86_64 14/20 2026-03-31T21:28:37.700 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : thrift-0.15.0-4.el9.x86_64 15/20 2026-03-31T21:28:37.703 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libnbd-1.20.3-4.el9.x86_64 16/20 2026-03-31T21:28:37.704 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : libpmemobj-1.12.1-1.el9.x86_64 17/20 2026-03-31T21:28:37.707 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : boost-program-options-1.75.0-13.el9.x86_64 18/20 2026-03-31T21:28:37.709 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : librabbitmq-0.11.0-7.el9.x86_64 19/20 2026-03-31T21:28:37.722 INFO:teuthology.orchestra.run.vm00.stdout: Erasing : librdkafka-1.6.1-102.el9.x86_64 20/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Running scriptlet: librdkafka-1.6.1-102.el9.x86_64 20/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 1/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 2/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 3/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 4/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 5/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 6/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librados2-2:20.2.0-721.g5bb32787.el9.x86_64 7/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 8/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 9/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 10/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 11/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 12/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rados-2:20.2.0-721.g5bb32787.el9.x86_64 13/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rbd-2:20.2.0-721.g5bb32787.el9.x86_64 14/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : python3-rgw-2:20.2.0-721.g5bb32787.el9.x86_64 15/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : qemu-kvm-block-rbd-17:10.1.0-16.el9.x86_64 16/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : rbd-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 17/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : rbd-nbd-2:20.2.0-721.g5bb32787.el9.x86_64 18/20 2026-03-31T21:28:37.784 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : re2-1:20211101-20.el9.x86_64 19/20 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 20/20 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout:Removed: 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: libarrow-9.0.0-15.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: libnbd-1.20.3-4.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: librados2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: librbd1-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: librgw2-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: python3-rados-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: python3-rbd-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: python3-rgw-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: qemu-kvm-block-rbd-17:10.1.0-16.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: rbd-fuse-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: rbd-nbd-2:20.2.0-721.g5bb32787.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: re2-1:20211101-20.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: thrift-0.15.0-4.el9.x86_64 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout: 2026-03-31T21:28:37.828 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:38.014 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: librbd1 2026-03-31T21:28:38.014 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:38.016 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:38.017 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:38.017 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:38.199 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: python3-rados 2026-03-31T21:28:38.199 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:38.201 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:38.202 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:38.202 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:38.365 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: python3-rgw 2026-03-31T21:28:38.365 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:38.367 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:38.367 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:38.367 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:38.532 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: python3-cephfs 2026-03-31T21:28:38.532 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:38.534 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:38.535 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:38.535 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:38.692 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: python3-rbd 2026-03-31T21:28:38.692 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:38.695 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:38.695 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:38.695 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:38.861 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: rbd-fuse 2026-03-31T21:28:38.861 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:38.863 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:38.864 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:38.864 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:39.032 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: rbd-mirror 2026-03-31T21:28:39.032 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:39.035 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:39.035 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:39.035 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:39.195 INFO:teuthology.orchestra.run.vm00.stdout:No match for argument: rbd-nbd 2026-03-31T21:28:39.195 INFO:teuthology.orchestra.run.vm00.stderr:No packages marked for removal. 2026-03-31T21:28:39.198 INFO:teuthology.orchestra.run.vm00.stdout:Dependencies resolved. 2026-03-31T21:28:39.198 INFO:teuthology.orchestra.run.vm00.stdout:Nothing to do. 2026-03-31T21:28:39.198 INFO:teuthology.orchestra.run.vm00.stdout:Complete! 2026-03-31T21:28:39.221 DEBUG:teuthology.orchestra.run.vm00:> sudo yum clean all 2026-03-31T21:28:39.341 INFO:teuthology.orchestra.run.vm00.stdout:56 files removed 2026-03-31T21:28:39.363 DEBUG:teuthology.orchestra.run.vm00:> sudo rm -f /etc/yum.repos.d/ceph.repo 2026-03-31T21:28:39.387 DEBUG:teuthology.orchestra.run.vm00:> sudo yum clean expire-cache 2026-03-31T21:28:39.543 INFO:teuthology.orchestra.run.vm00.stdout:Cache was expired 2026-03-31T21:28:39.543 INFO:teuthology.orchestra.run.vm00.stdout:0 files removed 2026-03-31T21:28:39.560 DEBUG:teuthology.parallel:result is None 2026-03-31T21:28:39.561 INFO:teuthology.task.install:Removing ceph sources lists on ubuntu@vm00.local 2026-03-31T21:28:39.561 DEBUG:teuthology.orchestra.run.vm00:> sudo rm -f /etc/yum.repos.d/ceph.repo 2026-03-31T21:28:39.585 DEBUG:teuthology.orchestra.run.vm00:> sudo mv -f /etc/yum/pluginconf.d/priorities.conf.orig /etc/yum/pluginconf.d/priorities.conf 2026-03-31T21:28:39.647 DEBUG:teuthology.parallel:result is None 2026-03-31T21:28:39.647 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-03-31T21:28:39.649 INFO:teuthology.task.clock:Checking final clock skew... 2026-03-31T21:28:39.649 DEBUG:teuthology.orchestra.run.vm00:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-03-31T21:28:39.703 INFO:teuthology.orchestra.run.vm00.stderr:bash: line 1: ntpq: command not found 2026-03-31T21:28:39.788 INFO:teuthology.orchestra.run.vm00.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-03-31T21:28:39.788 INFO:teuthology.orchestra.run.vm00.stdout:=============================================================================== 2026-03-31T21:28:39.788 INFO:teuthology.orchestra.run.vm00.stdout:^+ time.netzwerge.de 2 6 377 45 -1188us[-1188us] +/- 18ms 2026-03-31T21:28:39.788 INFO:teuthology.orchestra.run.vm00.stdout:^+ 172-104-154-182.ip.linod> 2 7 377 45 +2848us[+2848us] +/- 31ms 2026-03-31T21:28:39.788 INFO:teuthology.orchestra.run.vm00.stdout:^+ 185.252.140.125 2 6 377 43 -1751us[-1751us] +/- 21ms 2026-03-31T21:28:39.788 INFO:teuthology.orchestra.run.vm00.stdout:^* ntp.ntstime.org 2 7 377 46 +963us[ +947us] +/- 18ms 2026-03-31T21:28:39.788 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-03-31T21:28:39.790 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-03-31T21:28:39.790 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-03-31T21:28:39.792 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-03-31T21:28:39.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-03-31T21:28:39.796 INFO:teuthology.task.internal:Duration was 1821.649527 seconds 2026-03-31T21:28:39.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-03-31T21:28:39.799 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-03-31T21:28:39.799 DEBUG:teuthology.orchestra.run.vm00:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-03-31T21:28:39.874 INFO:teuthology.orchestra.run.vm00.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-03-31T21:28:40.155 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-03-31T21:28:40.155 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@vm00.local 2026-03-31T21:28:40.155 DEBUG:teuthology.orchestra.run.vm00:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-03-31T21:28:40.176 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-03-31T21:28:40.176 DEBUG:teuthology.orchestra.run.vm00:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-03-31T21:28:40.658 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-03-31T21:28:40.659 DEBUG:teuthology.orchestra.run.vm00:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-03-31T21:28:40.681 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-03-31T21:28:40.681 INFO:teuthology.orchestra.run.vm00.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-03-31T21:28:40.681 INFO:teuthology.orchestra.run.vm00.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-03-31T21:28:40.681 INFO:teuthology.orchestra.run.vm00.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-03-31T21:28:40.682 INFO:teuthology.orchestra.run.vm00.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-03-31T21:28:40.835 INFO:teuthology.orchestra.run.vm00.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 98.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-03-31T21:28:40.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-03-31T21:28:40.840 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-03-31T21:28:40.840 DEBUG:teuthology.orchestra.run.vm00:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-03-31T21:28:40.903 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-03-31T21:28:40.905 DEBUG:teuthology.orchestra.run.vm00:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-03-31T21:28:40.968 INFO:teuthology.orchestra.run.vm00.stdout:kernel.core_pattern = core 2026-03-31T21:28:40.983 DEBUG:teuthology.orchestra.run.vm00:> test -e /home/ubuntu/cephtest/archive/coredump 2026-03-31T21:28:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-03-31T21:28:41.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-03-31T21:28:41.041 INFO:teuthology.task.internal:Transferring archived files... 2026-03-31T21:28:41.041 DEBUG:teuthology.misc:Transferring archived files from vm00:/home/ubuntu/cephtest/archive to /archive/kyr-2026-03-31_11:18:10-rados-tentacle-none-default-vps/4348/remote/vm00 2026-03-31T21:28:41.041 DEBUG:teuthology.orchestra.run.vm00:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-03-31T21:28:41.106 INFO:teuthology.task.internal:Removing archive directory... 2026-03-31T21:28:41.106 DEBUG:teuthology.orchestra.run.vm00:> rm -rf -- /home/ubuntu/cephtest/archive 2026-03-31T21:28:41.163 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-03-31T21:28:41.166 INFO:teuthology.task.internal:Not uploading archives. 2026-03-31T21:28:41.166 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-03-31T21:28:41.168 INFO:teuthology.task.internal:Tidying up after the test... 2026-03-31T21:28:41.168 DEBUG:teuthology.orchestra.run.vm00:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-03-31T21:28:41.221 INFO:teuthology.orchestra.run.vm00.stdout: 8532145 0 drwxr-xr-x 2 ubuntu ubuntu 6 Mar 31 21:28 /home/ubuntu/cephtest 2026-03-31T21:28:41.222 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-03-31T21:28:41.228 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/{bluestore/{alloc$/{avl} base mem$/{normal-2} onode-segment$/{256K} write$/{random/{compr$/{no$/{no}} random}}}} rados supported-random-distro$/{centos_latest}} duration: 1821.6495265960693 flavor: default owner: kyr success: true 2026-03-31T21:28:41.228 DEBUG:teuthology.report:Pushing job info to http://localhost:8080 2026-03-31T21:28:41.248 INFO:teuthology.run:pass